Comments: I'm not comfortable with the lack of _full_ test automation, including verification of the final installed image. If the final answer is that we must have manual testing, I think QE will need to own that.
I think we should have PIT runs. Why no memory leak testing? I don't see where it says to verify that the installed image works...does it? On Aug 25, 2009, at 5:07 AM, angela wrote: > Hi Guys, > > A draft of the test plan for Text Installer test plan for 2010.02 is > attached. Feedback is appreciated. > > I intend to have final version at Aug. 31. Please give feedback to me > in this week. > > Thanks, > Angela > > Text Installer Test Plan > > There is no ARC ID for this project yet. > > Introduction > > Authors > > Name > > Email > > Extension / Region > > Angela Li > > Angela.Li at Sun.Com > > x82224 (2) > > (1) US/Canada/South America or (2) Europe/Middle East/Africa/Asia > > > > > Test Sponsor > > The text installer will be delivered via the OpenSolaris 1002 > release, not by checking into the Install consolidation. As a > result, the C-team process (including the assignment of a test > sponsor) is not being followed for the OpenSolaris 1002 delivery. A > test sponsor will be requested at such time that the text installer > is nearing integration into a consolidation. > > Test Plan Approvals > > Title > > Name > > Version > > Date > > Development Lead > > X.X > > YYYY-MM-DD > > Development Manager > > X.X > > YYYY-MM-DD > > QE management > > > > > > > > > External Open Source Test Software > > Does this project use any external open source test software? No > > Revision History > > Date > (yyyy-mm-dd) > > Revision > > Comments > > Approval/Status > > 2009-08-17 > > 0.1 > > Initial draft for QE team review > > submitted to QE team and I-team > > 2009-08-25 > > 0.2 > > Incorporate I-team comments > Updated Schedules and Milestones section according to new schedule > version text_installer_082409.pdf > > submitted to caiman-discuss > > > > > > > > > > > > > > > > > > > > > > > > > > > > > References > > Pointer to the ON test plan: > Pointer to the C-team project plan: N/A > Pointers to other documentation useful in understanding the test > program: > OpenProj Plan: 08/24/2009: > https://securewiki.sun.com/download/attachments/24052578/text_installer_082409.pdf?version=1 > Functional Spec: > http://www.opensolaris.org/os/project/caiman/TextInstallerProject/text_install_spec_v3.txt > Project Documentation: > http://www.opensolaris.org/os/project/caiman/TextInstallerProject/ > UI Spec: > http://xdesign.sfbay.sun.com/projects/solaris/subprojects/install/design/screensText2.html > OpenSolaris 1002 Schedule: > http://infoshare.sfbay/twiki/bin/view/Main/OpenSolaris201002Schedule > > Staffing > > Name > > Role > > % Commitment > > Duration > > Angela Li > > Lead > > 100% > > 6 months > > Jason Zhao > > Engineer > > 50% > > 6 months > > Rokh > > Engineer > > 50% > > 6 months > > > > > Glossary > > Term > > Definition > > curses > > a library of terminal I/O functions that provide the user with a > terminal-independent method of updating character screens with > reasonable optimization. > > nCurses (new curses) > > a freely redistributable emulation of the curses library in System V > Release 4, which also provides many useful extentions/enhancements > over SVR4 Curses. > > extended partition > > a primary partition (x86) that has been divided up into logical > partitions as a means of creating more partitions than the four that > would otherwise be possible. Only one primary partition can be used > as an extended partition, and it can be created from any of the > primary partitions. > > Transfer Module > > Software component on the DVD that copies the OS from the DVD to > hard disk. > > > > > Test Program Summary > > Interactive Text-based Installer provides interactive text-based > install capability for SPARC and x86. The primary target is server > installations. It also enables OpenSolaris installation in systems > without a graphics card and networking. > > The major components of text installer are as the following: > > nCurses UI equivalent to current GUI installer > Static IP configuration > VTOC Slice management > extended partitions > > For Static IP Configuration, it assumes the use of an existing > utility or method for configuring network information (e.g. SMF > profiles). If no such utility exists, it will have to be developed. > > For extended partition management and slice management and > preservation, it is assumed that the necessary library code will > already be available. The text installer's role in these areas is > purely that of the UI. > > Language and Keyboard layout are selected prior to starting the text > installer (similar to how the liveCD asks for language and keyboard > prior to starting the gdm desktop environment). The language > selection is used as the default during the text install, but can be > modified for the installed system. > > The test program will include manual test and automated test. The > manual test will cover small menu of choices test, full installation > covering the test matrix defined in table 6.4.1, post-verification > after reboot. The automated test will run automated test suite > (textui) on X86, SPARC, xVM, Virtual Box, LDOM. > > QE team will focus on automated test suite development, and the text > installer automatic test suite will be delivered at the same time > when the project integrate. > > During the period prior to the availability of text installer > automatic test suite (textui), manual test will be done by I-Team. > As I-team have less hardware resources than described in table > 6.4.1, so only basic manual test can be done before integration. > > The text installer automatic test suite will be under TET-CTI test > environment whenever possible, although there are expected to be a > few tests that do not lend themselves to automation that will be > executed manually. > > Testing will focus on the text installer user interface (UI). > Assertions will be developed for the functionality of the text > installer user interface (UI). The automatic test suite will use > EXPECT to capture the content of text installer UI screen. > > Testing will not focus the stress test, performance test and > conformance test. > > Operational Factors > > Assumptions > > Assumption #1: The QE will focus on automated test suite > development, so bits before OpenSolaris 1002 test builds will not be > tested by QE team manually. > > Assumption #2: For automated test requirement, all the text > installer's output, especially strings in the screens and pop-up > windows, could be captured by EXPECT, EXPECT could do input to the > text installer. > > Assumption #3: The fully internationalization support testing will > not be covered by the test program. > > Dependencies > > Dependency #1: The testable bits will be provided as early as > possible before OpenSolaris 1002 test build 1 for QE develop > automated test suite. > > Risks > > Risk #1: > > Description: Any difference between the text installer's design and > implementation may lead the automated test suite change. > Likelihood of Occurrence: High > Mitigation and Contingency Plan: Tests will be developed most > independent with the text installer's implementation. > Risk #2: > > Description: The change in Extended Partition project shedule and > libbe schedule will also impact the text installer automated test > suite development schedule. > Likelihood of Occurrence: High > Mitigation and Contingency Plan: Communicate and re-schedule. > Risk #2: > > Description: The automated test suite code review may not be > finished in 4 days as to the reviewer's schedule > Likelihood of Occurrence: Middle > Mitigation and Contingency Plan: Communicate and begin code review > as earlier as possible > > > > Test Development Information > > Test Development > > Test Name > > Automated or Manual > > Test Type > > Project > Component > > Description > > textui > > automated > > functional > > Text installer UI > > Verity the text installer options and the ability to perform an > actual installation > > > > > Areas of Testing > > Functional Testing > > Verify all options in the text installer UI. > > The text installer will have the following screens: > > Intro/Navigation > Disk Selection > Fdisk Partition List (x86 only) > Fdisk Partition Management (x86 only) > VTOC Slice Management > Static IP Configuration > Time Zone (region) > Time Zone (locations) > Time Zone Selection > Date/Time Setting > Locale - Default Language Support > Root password and User Account > Summary and Pre-Installation Review > Installation Progress > Installation Results > Help screens > > Each action on each screen will be covered, including Continue, > Back, Help, and any specific action on specific screens. > > Valid user input and invalid user input will be carried out to test > error handling. > Regression Testing > > No regression testing. > > Conformance Testing > > The text installer does not have to conform to any standards so no > conformance testing is planned. > > Stress/Robustness Testing > > No stress and robustness testing. The installer runs on an OS image > with only one user account enabled. > > Performance Testing > > The text installer will provide comparable install-time performance > to the GUI installer (for x86), relative to the size of the image. > No performance testing, but the overall installation time will be > collected to say whether it meets the text installer's function > specification. > Memory Leak Testing > > No memory leak testing. > > Required Solaris Feature Testing > > The installer runs on a Solaris microroot using a DVD as the boot > media and a small ramdisk as the only writable space, so the normal > required feature testing does not apply during the install process. > When booted an installation on disk, some required feature testing > can be done: > > Zones Testing: N/A > > ZFS Testing: The functionality of the zpool command with regards to > the root file system will be verified. > > Trusted Extensions Testing: N/A > > xVM Guest OS Testing: Only HVM will be tested. > > LDOM Guest OS Testing: Will attempt to install Solaris under LDOM. > > virtual box: Will attempt to install Solaris under virtual box. > Currently the latest VB version is 3.0.4. > > Interoperability Testing > > Installs on systems with pre-existing installations of other OS > types (Windows, Linux) will be performed to verify that Solaris can > be installed alongside other OS's. > > Documentation Testing > > N/A > > Internationalization Testing > > Only basic verification that the specified locale and language > support are installed will be performed. > > Additional Testing (e.g. BCP, EA Programs) > > Low-memory testing: Install will be performed on a system with the > minimum (512MB) supported memory configuration. > > Testing Not Covered > > Any testing beyond verifying the installer UI itself, such as > Extended Partition, Partition Management, Slice Management, Static > IP Management. > > The Transfer Module for transfer of contents from the DVD to hard > disk, and basic verification of the installed OS is beyond the scope > of this test plan. Targeted testing of bundled Solaris components or > applications (X, Firefox, etc.) are left to other groups as > coordinated by the overall OpenSolaris 1002 test lead. > > Test Execution > > Hardware Test Configurations > > The following HW platforms will be tested in various combinations: > > sun4u , sun4v, i386, amd64 > low memory configurations (in addition to normal memory > configurations) > VM tech: xVM, Virtual Box, LDOM > > Software Test Configurations > > Mostly N/A, as the install image containing Text Installer controls > the version of Solaris and all applications/utilities that will be > installed. Testing of software beyond the installer itself and > verification of basic OS functionality is beyond the scope of > Solaris QE's qualification. > > Extrapolation Strategy (if applicable) > > N/A > > Pre-Integration PIT Run > > Is a pre-integration PIT run required? (REQUIRED, RECOMMENDED, > NONE)? NONE - the text installer bits will be integrating into > Slim_source, not into Solaris consolidations. > If a pre-integration PIT run is required or recommended, should the > run be the OS PIT, Net PIT or both? > > Test Execution Matrix > > Table 6.4.1: Machine HW/SW Configurations > > # > > Model > > Type > > Host Name > > Arch > > Memory > > Kernel > > Test Installation Configuration ( > - previous software installed > - disk type > - partition/slice management > - ip management > - UFS/ZFS > - user) > > Virtualization > > 1 > > Dell270 > > desktop > > raindoctor > > i386 > > 512M > > 32 > > - N/A of previous software installed > - IDE disk(80G): with linux partition, solaris partion and extended > partition > - A slice in a partition > - Manually IP > - root user > > N/A > > 2 > > Toshiba M10 > > laptop > > > > amd64 > > 4G > > 64 > > - N/A of previous software installed > - SATA disk(1600G): with windows partition, extented partition > (Solaris logical partition) + USB disk > - Whole partition > - Automatically IP > - non-root uesr > > N/A > > 3 > > PC - Core2 > > desktop > > rainout > > > > 2G > > 64 > > - S10Update ZFS > - SATA disk(80G) and +1TB disk: install on +1TB > - Whole partition > - None IP > - non-root user > > N/A > > 4 > > Sun Ultra 40 M2 > > workstation > > rainboot > > amd64 > > 4G > > 64 > > - N/A of previous software installed > - SATA disk (250G) > - Whole disk > - None IP > - root user > > xVM > > 5 > > Sun Ultra 40 > > server > > rainday > > amd64 > > 1G > > 64 > > - N/A of previous software installed > - SCSI disk > - Whole disk > - None IP > - root user > > Virtual Box > > 6 > > Sun Fire V240 > > server > > flowerboat > > sun4u > > 8G > > 64 > > - S10Update ZFS > - SCSI disk > - A slice in a disk > - Automatically IP > - root user > > N/A > > 7 > > Sun Blade 150 > > server > > rainbox > > sun4u > > 512M > > 64 > > - N/A of previous software installed > - IDE disk(80G) > - whole disk > - None IP > - non-root user > > N/A > > 8 > > Sun T2000 > > server > > redmoon > > sun4v > > 8G > > 64 > > - N/A of previous software installed > - SCSI disk (70G) and +1TB disk > - A slice in a disk > - Manually IP > - non-root user > > N/A > > 9 > > Sun T2000 > > server > > earthcolor > > sun4v > > 3G > > 64 > > - N/A of previous software installed > - SCSI disk (70G) > - Whole disk > - None IP > - root user > > LDOM > > > > > Table 6.4.2: Test Suite Execution > > # > > Test Suite Execution > > 1 > > textui > > 2 > > > > > > > Schedules and Milestones > > Milestone > > Original Date > (yyyy-mm-dd) > > Actual Date > (yyyy-mm-dd) > > Test Plan First Draft > > 2009-08-21 > > 2009-08-17 > > Test Plan I-team Approval > > 2009-08-28 > > > > Test Plan QE Management Approval > > 2009-08-28 > > > > Text Installer Design Completed > > 2009-08-31 > > > > Test Assertions Completed and Approved by I-team > > 2009-09-04 > > > > Automated Test Suite Development Completed > > 2009-11-13 > > > > Automated Test Suite Code Review Completed > > 2009-11-19 > > > > Automated Test Suite Integration into STC2 > > 2009-11-20 > > > > Text Installer Integrated > > 2009-12-04 > > > > OpenSolaris 1002 Test Build 1 > > 2009-12-03 > > > > Initial test execution complete of all tests defined in this project > > 2009-12-16 > > > > OpenSolaris 1002 Test Build 2 > > 2009-12-17 > > > > OpenSolaris 1002 Test Build 3 > > 2010-01-14 > > > > OpenSolaris 1002 Test Build 4 > > 2010-01-28 > > > > Final test execution cycle > > 2010-02-12 > > > > Integration into ON PIT > > 2010-03-25 > > > > > > > Post-Integration Testing Information > > Test Suite Integration Requirements > > Test/Test Suite Name > > Source Integration Location > > Target Integration Date > > Comments > > textui > > STC 2.0 > > 2009-11-20 > > > > > > > > > > > > > > > Test Suite Execution Integration Requirements > > Test/Test Suite Name > > Execution Group > > Comments > > textui > > PIT > > > > > > > > > > > > > Other Post-Integration Requirements > > N/A > > Solaris Update Release (if applicable) > > OpenSolaris 1002 constitutes its own release of sorts. The various > Indiana components will eventually end up in Solaris consolidations, > but will be handled as separate qualifications and so are not > covered by this test plan. > > > > > _______________________________________________ > caiman-discuss mailing list > caiman-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/caiman-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/caiman-discuss/attachments/20090825/9513b7bd/attachment.html>
