Hi, Ethan and Sundar, I modified some test cases and improved test report format, and also added some comments to the failed cases in report.
Please review if the new report is more comfortable: http://opg-qe.central.sun.com/wiki/index.php/Install_ai:osol-0906-b2 In the report are our 2nd week test results on build 111, testing was against a SPARC AI server. Also please see my reply inline. Ethan Quach ??: > Jeffrey, > > > Jeffrey Huang wrote: >> Hi, Ethan, >> >> Ethan Quach ??: >>> Jeffrey, >>> >>> I'm filling in for Sundar this week, so just catching up. I have >>> a couple of questions/comments about the test results ... >>> >>> 1. What does the result code of "Uninitialized" mean? Are >>> these failures we need to look at and address as bugs? >>> What does the result code of "Other" mean? >> >> Thanks very much to look into the report details. >> The "Uninitialized" of this test suite happened when adding a test >> manifest to >> AI service, caused by 2 reasons: >> 1. product bugs that failed to add certain manifest to service. >> 2. Test env of last execution is not very well cleaned >> Now, I am manually verifying above cases if "Uninitialized" happened, >> I will improve the >> clean up process so only manifest bugs will cause uninitialized. > > It would good to know the cases where (1) happened as well. Yes, now the error output is caught in testlog if failed to add manifest, then from logged error message we can tell if it is a product bug. > >> >> "Other" caused by 2 reasons: >> 1. Test case aborted by unexpected reason. ( We will improve >> individual case to avoid them) >> 2. Disk partition features not supported on SPARC platform. These >> cases should be reported >> as Unsupported, unfortunately, the TET/CTI report harness can not >> identify it.) > > Thanks for the explanation. > >> >>> >>> 2. In the summary.html Details section, it would be nice if the >>> Fail/Other column was split out into three columns -- >>> Fail, Unintialized, and Other so that we can quickly find the >>> kind we're looking for. >> >> Yes, 3 columns would be much better, we plan to improve the report in >> 2 two ways: >> 1. Enhance TET/CTI report harness to list category in different >> columns and also identify >> unsupported cases. >> 2. We also manually add notation to the failed cases with further >> detailed info in next >> test report. > > > I had one more question below from the previous mail ... > >>> >>> >>>> BTW, in build 111, I see AI manifest supports install package other >>>> than >>>> default build in repo, >>>> seems this feature is not from the design doc, also it is not >>>> covered by >>>> our test plan and test suite right >>>> now. >>>> >>> >>> Need clarification. Are you referring to the list of packages which >>> are now in the default manifest? i.e: >>> >>> <ai_install_packages> >>> <pkg name="SUNWcsd"/> >>> <pkg name="SUNWcs"/> >>> <pkg name="babel_install"/> >>> <pkg name="entire"/> >>> </ai_install_packages> >>> <ai_uninstall_packages> >>> <pkg name="babel_install"/> >>> <pkg name="slim_install"/> >>> </ai_uninstall_packages> >>> >>> No, the feature I referred here is to install package with a specified build other than default in IPS repo. like following, instead of writing: <ai_install_packages> seems manifest also can be written as: <pkg_name="entire at 0.5.11-0.build#/> I see above in default.xml in some early build in 111 last week, but from the SUNWinstalladm-tool I just installed on Tuesday, they are gone. Then I am not sure it is a new feature that be officially delivered in 09.06? Thanks Jeffrey >>> >>> thanks, >>> -ethan >>> >> -- Great Thanks & Best Regards --- Jeffrey Huang SWAN: x51683, Ext: 0086-10-62673683 Solaris QE Group Email: jeffrey.huang at sun.com China Engineering & Research Institute, Sun Microsystems 7/F Chuang Xin Plaza, Tsinghua Science Park, Beijing China
