On 8/28/06, Ivanov, Alexey A <[EMAIL PROTECTED]> wrote:
>-----Original Message----- >From: Richard Liang [mailto:[EMAIL PROTECTED] >Sent: Friday, August 25, 2006 7:18 PM >To: harmony-dev@incubator.apache.org >Subject: [classlib][TestNG] groups of Harmony test > >Hello All, > >Now let's talk about the TestNG groups. I have read the related threads >which posted by George, Vladimir Ivanov and Alexei Zakharov. All of them >are good discussion about TestNG groups. > >IMHO, we may define Harmony test groups according the following 4 >dimensions: > >1) [Platform] os.any, os.<platform id> > *os.any* - group of tests which pass on any platform. IMHO, most of our >tests should be in this group. > *os.<platform id>* - group of tests which are designed for one specific >platform. A test may be in more than one of the groups. e.g., >@Test(groups={"os.win.IA32", "os.linux.IA32"}) > > ** os.any and os.<platform id> are mutually exclusive, that is, >tests in os.any group should not be in os.win.IA32. > >2) [Test state] state.broken, state.broken.<platform id> > *state.broken* - group of tests which fail on every platform, because >of bugs of tests or implementation. We need to fix the bugs of tests or >implementation to make them pass. > *state.broken.<platform id>* - groups of test which only fail on one >specific platform. A test may be in more than one of the groups. e.g., >@Test(groups={"state.broken.linux.IA32", "os.broken.linux.IA64"}) > > **state.broken.<platform id> group may be used as a convenient way >to indicate that a test is platform-specific. e.g., If we support 10 >platforms, and one test are designed for 9 platforms except for MacOS, >instead of list 9 os.<platform id>, we can just use state.broken.MacOS > >3) [Test type] type.api, type.impl > *type.api* - group of tests which are tests for APIs in the Java >Specification > *type.impl* - groups of tests which are tests for Harmony-specific >implementation > > ** type.api and type.impl are also mutually exclusive. > >4) [Test Level] level.unit, level.integration, level.system, >level.stress, etc. (Levels of Test refer to the increase in complexity >as moving through test cycle. ) > ** A test may be in more than one of the groups. > ** In fact, some tests such as System tests are the verification of >the entire system. Maybe we'll put them into a separate project. e.g., >harmony/enhanced/SVT (System Verification Test). 5) [Environment] env.display, env.headless To distinguish AWT and Swing tests which need a display to run, and those which don't, as Mark proposed [1].
Will display option be passed manually as an argument to TestNG, or detected automatically when running test? Regards,
Alexey. [1] http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200608.mb ox/[EMAIL PROTECTED] > >If we want to run all the unit test for APIs on windows, we may use >TestNG groups to select the tests: > <groups> > <run> > <include name="os.any" /> > <include name="type.api" /> > <include name="os.win.IA32" /> > <exclude name="state.broken" /> > <exclude name="state.broken.win.IA32" /> > </run> > </groups> > > >Well, I think our most of existing tests are in the groups of {"os.any", >"type.api", "level.unit"}, and I have asked TestNG to add a new option >"-groups" for its JUnitConverter which allow us to specify the test >groups when migrate from JUnit test to TestNG test. > >Thanks for reading so far, and I will highly appreciate your comments or >suggestion. ;-) > >-- >Richard Liang >China Software Development Lab, IBM > > > >--------------------------------------------------------------------- >Terms of use : http://incubator.apache.org/harmony/mailing.html >To unsubscribe, e-mail: [EMAIL PROTECTED] >For additional commands, e-mail: [EMAIL PROTECTED] -- Alexey A. Ivanov Intel Middleware Product Division --------------------------------------------------------------------- Terms of use : http://incubator.apache.org/harmony/mailing.html To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
-- Andrew Zhang China Software Development Lab, IBM
