George Harley wrote: > Hi Geir, > > As you may recall, a while back I floated the idea and supplied some > seed code to define all known test failing test methods in an XML file > (an "exclusions list") that could be used by JUnit at test run time to > skip over them while allowing the rest of the test methods in a class to > run [1]. Obviously I thought about that when catching up with this > thread but, more importantly, your comment about being reluctant to have > more dependencies on JUnit also motivated me to go off and read some > more about TestNG [2]. > > It was news to me that TestNG provides out-of-the-box support for > excluding specific test methods as well as groups of methods (where the > groups are declared in source file annotations or Javadoc comments). > Even better, it can do this on existing JUnit test code provided that > the necessary meta-data (annotations if compiling to a 1.5 target; > Javadoc comments if targeting 1.4 like we currently are). There is a > utility available in the TestNG download and also in the Eclipse support > plug-in that helps migrate directories of existing JUnit tests to TestNG > by adding in the basic meta-data (although for me the Eclipse version > also tried to break the test class inheritance from > junit.framework.TestCase which was definitely not what was required). > > Perhaps ... just perhaps ... we should be looking at something like > TestNG (or my wonderful "exclusions list" :-) ) to provide the > granularity of test configuration that we need. > > Just a thought.
How 'bout that ;) geir > > Best regards, > George > > [1] http://issues.apache.org/jira/browse/HARMONY-263 > [2] http://testng.org > > > > Geir Magnusson Jr wrote: >> Alexei Zakharov wrote: >> >>> Hi, >>> +1 for (3), but I think it will be better to define suite() method and >>> enumerate passing tests there rather than to comment out the code. >>> >> >> I'm reluctant to see more dependencies on JUnit when we could control at >> a level higher in the build system. >> >> Hard to explain, I guess, but if our exclusions are buried in .java, I >> would think that reporting and tracking over time is going to be much >> harder. >> >> geir >> >> >>> 2006/6/27, Richard Liang <[EMAIL PROTECTED]>: >>> >>>> Hello Vladimir, >>>> >>>> +1 to option 3) . We shall comment the failed test cases out and add >>>> FIXME to remind us to diagnose the problems later. ;-) >>>> >>>> Vladimir Ivanov wrote: >>>> >>>>> I see your point. >>>>> But I feel that we can miss regression in non-tested code if we >>>>> exclude >>>>> TestCases. >>>>> Now, for example we miss testing of >>>>> >>>> java.lang.Class/Process/Thread/String >>>> >>>>> and some other classes. >>>>> >>>>> While we have failing tests and don't want to pay attention to these >>>>> failures we can: >>>>> 1) Leave things as is – do not run TestCases with failing tests. >>>>> 2) Split passing/failing TestCase into separate "failing TestCase" and >>>>> "passing TestCase" and exclude "failing TestCases". When test or >>>>> implementation is fixed we move tests from failing TestCase to passing >>>>> TestCase. >>>>> 3) Comment failing tests in TestCases. It is better to run 58 tests >>>>> instead >>>>> of 0 for String. >>>>> 4) Run all TestCases, then, compare test run results with the 'list of >>>>> known >>>>> failures' and see whether new failures appeared. This, I think, is >>>>> >>>> better >>>> >>>>> then 1, 2 and 3, but, overhead is that we support 2 lists - list of >>>>> >>>> known >>>> >>>>> failing tests and exclude list where we put crashing tests. >>>>> >>>>> Thanks, Vladimir >>>>> On 6/26/06, Tim Ellison <[EMAIL PROTECTED]> wrote: >>>>> >>>>>> Mikhail Loenko wrote: >>>>>> >>>>>>> Hi Vladimir, >>>>>>> >>>>>>> IMHO the tests are to verify that an update does not introduce any >>>>>>> regression. So there are two options: remember which exactly >>>>>>> >>>> tests may >>>> >>>>>> fail >>>>>> >>>>>>> and remember that all tests must pass. I believe the latter one is >>>>>>> >>>>>> a bit >>>>>> >>>>>>> easier and safer. >>>>>>> >>>>>> +1 >>>>>> >>>>>> Tim >>>>>> >>>>>> >>>>>>> Thanks, >>>>>>> Mikhail >>>>>>> >>>>>>> 2006/6/26, Vladimir Ivanov <[EMAIL PROTECTED]>: >>>>>>> >>>>>>>> Hi, >>>>>>>> Working with tests I noticed that we are excluding some tests just >>>>>>>> because >>>>>>>> several tests from single TestCase fail. >>>>>>>> >>>>>>>> For example, the TestCase 'tests.api.java.lang.StringTest' has 60 >>>>>>>> tests and >>>>>>>> only 2 of them fails. But the build excludes the whole TestCase >>>>>>>> >>>>>> and we >>>>>> >>>>>>>> just >>>>>>>> miss testing of java.lang.String implementation. >>>>>>>> >>>>>>>> Do we really need to exclude TestCases in 'ant test' target? >>>>>>>> >>>>>>>> My suggestion is: do not exclude any tests until it crashes VM. >>>>>>>> If somebody needs a list of tests that always passed a separated >>>>>>>> target can >>>>>>>> be added to build. >>>>>>>> >>>>>>>> Do you think we should add target 'test-all' to the build? >>>>>>>> Thanks, Vladimir >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>> --------------------------------------------------------------------- >>>> >>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html >>>>>>> To unsubscribe, e-mail: [EMAIL PROTECTED] >>>>>>> For additional commands, e-mail: >>>>>>> >>>> [EMAIL PROTECTED] >>>> >>>>>>> >>>>>> -- >>>>>> >>>>>> Tim Ellison ([EMAIL PROTECTED]) >>>>>> IBM Java technology centre, UK. >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html >>>>>> To unsubscribe, e-mail: [EMAIL PROTECTED] >>>>>> For additional commands, e-mail: >>>>>> [EMAIL PROTECTED] >>>>>> >>>>>> >>>>>> >>>> -- >>>> Richard Liang >>>> China Software Development Lab, IBM >>>> >>> >>> >> >> --------------------------------------------------------------------- >> Terms of use : http://incubator.apache.org/harmony/mailing.html >> To unsubscribe, e-mail: [EMAIL PROTECTED] >> For additional commands, e-mail: [EMAIL PROTECTED] >> >> >> > > > --------------------------------------------------------------------- > Terms of use : http://incubator.apache.org/harmony/mailing.html > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > > --------------------------------------------------------------------- Terms of use : http://incubator.apache.org/harmony/mailing.html To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
