On 7 May 2013 22:00, Rob Weir <[email protected]> wrote: > On Tue, May 7, 2013 at 3:42 PM, janI <[email protected]> wrote: > > On 7 May 2013 21:31, Rob Weir <[email protected]> wrote: > > > >> On Tue, May 7, 2013 at 11:46 AM, V Stuart Foote <[email protected] > > > >> wrote: > >> > Shenfeng, Yuzhen, Liuping, Rob > >> > > >> > Took a moment and browsed through the TestLink test cases for the Full > >> Regression and earlier test plans against the trunk and sidebar branch. > >> > > >> > Unless I completely missed seeing it, one noticeable omission from the > >> test cases in general is a lack keyboard based navigation and use of > >> accelerators. Would expect that to be a significant component of > >> functional testing. > >> > > >> > Only testing with mouse and menu based usage is unfortunately a way to > >> miss the framework needed for accessibility and Assistive Technology > >> support and impact all the supported OS builds. Not just the work on the > >> IA2 branch. > >> > > >> > >> So what would you recommend? Is this something you can help with, on > >> the test definition or execution side? > >> > >> My naive approach would be, "Make sure you can perform all test cases > >> with just the keyboard". Or is there a more targeted way to do this? > >> A regression test (as opposed to a full functional test) tries to be > >> smart about coverage and focuses on the areas that have changed in the > >> most recent version. In other words, it puts the effort where the > >> risk is greatest. > >> > > > > Sounds like a good approach to me "just with the keyboard". > > > > I hope that for all automated testing we only use "full functional test". > > With human resources involved regression testing is understandable. > > > > The regression test is a manual test, not an automated one. Automated > tests are "smoke tests" and have even less coverage than the > regression tests. So automation, at least what we have now, is not > going to solve this problem. > > > > >> > >> So, given what we know about AOO 3.4.1 keyboard support, and knowing > >> what was changed in AOO 4.0, my guess would be the area to focus on > >> would be the sidebar, yes? > >> > >> Translations are another area where I bet there is risk introduced. > >> It is always possible for an accelerator collision to be introduced > >> accidentally during translation. And the translations come in rather > >> late in the process, after most testing. I wonder if we have any > >> automated ways of detecting collisions? > >> > > > > if/when we have keyboard automated testing, it would be an easy job to > > write a script that maps the keyboard and accellerators to any language. > > That would allow us to run the same tests in any language. > > > > I was hoping for something closer to present capabilities. For > example, accelerators are defined in resource files, yes? It should > be possible to analyze the resources files directly to see if there is > a collision within any menu item. In theory. >
that is not just theory, I worked on a package earlier that did such an analysis on PO files...actually it is quite easy. The benefit of doing it on PO files is the consistency check (accelletor for a given menu item, e.g. Open is identical in all product parts). > > > Based on my experience from international software, menus and > accellerators > > are common translation problems. > > > > Yes. > > > > > > >> > >> Any other areas that would make sense to focus on? > >> > > > > Installation with a keyboard only, or automated installation > > > > I didn't think the install UI changed in 4.0, did it? > > When I say "focus" I'm reasoning like this: > > 1) No product of the size and complexity of AOO is able to be fully > tested, in any formal sense (line coverage, path coverage, etc.) > that is a statement...I have worked on bigger packages, where we had next to full coverage (all automated), so the size is not the problem, our restricted resources are, which is why we in the middleterm should focus on automating as much a possible. > 2) So we rely on a good sample of possible test cases that give good > overall coverage, combined with more focused testing on areas where we > think there is higher risk because of recent code changes. > for now that is the only thing we can do. > > 3) There are many areas that are important to specific users: > performance, running on a server, users with bidirectional writing > scripts, users with East Asian writing styles, users on older > versions of Windows versus newest versions of Linux, etc., And > keyboard use as well. We can't do every test in every combination. > (What about Windows 8 Hebrew users without a mouse?) So we need to > figure out the best combination of tests that give good overall > coverage. > +1 > > 4) To make the problem tractable areas that have not changed in AOO > 4.0 will get less test coverage. Not the same as no coverage, since > bugs can have strange interactions. But lacking 100% automated test > suites with perfect coverage, infinite volunteers or infinite time, we > need to figure out where to spend the time to find the most critical > bugs. Where are those bugs? I'm thinking they are not in the > install UI. But who knows... > +1 jan I. > > -Rob > > > rgds > > Jan I. > > > >> > >> Regards, > >> > >> -Rob > >> > >> > >> > Stuart > >> > San Antonio, Texas > >> > > >> > > >> > ________________________________________ > >> > From: Shenfeng Liu [[email protected]] > >> > Sent: Tuesday, May 07, 2013 8:28 AM > >> > To: [email protected]; [email protected] > >> > Subject: Start AOO 4.0 Full Regression Test > >> > > >> > Yu Zhen & Liu Ping, > >> > Thanks very much for your help to prepare for the test guidance! > >> > > >> > Now we have the latest dev snapshot build here: > >> > > >> > https://cwiki.apache.org/confluence/display/OOOUSERS/Development+Snapshot+Builds#DevelopmentSnapshotBuilds-AOOSnapshotfullsets > >> > And we have the "AOO 4.0 Full Regression Test" test plan created in > >> > Testlink: http://aootesting.adfinis-sygroup.org/index.php . > >> > And we have test guidance here: > >> > > >> > https://cwiki.apache.org/confluence/display/OOOUSERS/AOO+Full+Regression+Test+Guidance > >> > and here: > >> > > >> > https://cwiki.apache.org/confluence/display/OOOUSERS/AOO+Defect+Verification+Guidance > >> > > >> > So I suggest we start the AOO 4.0 Full Regression Test now !!!!! > >> > > >> > The test focus of this test circle include the following: > >> > (1) Basic Function Verification > >> > (2) MS Interoperability Test > >> > (3) 2nd phase of sidebar testing > >> > (4) Defect verification > >> > > >> > We will add more test cases (e.g. the Upgrade Check) in later > circle. > >> > > >> > We already got some volunteers replied in mail list. Thanks very > much! > >> > Will assign test cases in Testlink and send mail to you one by one. > >> > And looking forward more people to join and help the testing! > >> > > >> > - Shenfeng (Simon) > >> > > >> > > >> > --------------------------------------------------------------------- > >> > To unsubscribe, e-mail: [email protected] > >> > For additional commands, e-mail: [email protected] > >> > > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: [email protected] > >> For additional commands, e-mail: [email protected] > >> > >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
