George Harley wrote: > Nathan Beyer wrote: >> Two suggestions: >> 1. Approve the testing strategy [1] and implement/rework the modules >> appropriately. >> 2. Fix the tests! >> >> -Nathan >> >> [1] >> http://incubator.apache.org/harmony/subcomponents/classlibrary/testing.html >> >> > > Hi Nathan, > > What are your thoughts on running or not running test cases containing > problematic test methods while those methods are being investigated and > fixed up ? >
That's exactly the problem. We need a clear way to maintain and track this stuff. geir > > Best regards, > George > > >> >> >>> -----Original Message----- >>> From: Geir Magnusson Jr [mailto:[EMAIL PROTECTED] >>> Sent: Tuesday, June 27, 2006 12:09 PM >>> To: harmony-dev@incubator.apache.org >>> Subject: Re: [classlib][testing] excluding the failed tests >>> >>> >>> >>> George Harley wrote: >>> >>>> Hi Geir, >>>> >>>> As you may recall, a while back I floated the idea and supplied some >>>> seed code to define all known test failing test methods in an XML file >>>> (an "exclusions list") that could be used by JUnit at test run time to >>>> skip over them while allowing the rest of the test methods in a >>>> class to >>>> run [1]. Obviously I thought about that when catching up with this >>>> thread but, more importantly, your comment about being reluctant to >>>> have >>>> more dependencies on JUnit also motivated me to go off and read some >>>> more about TestNG [2]. >>>> >>>> It was news to me that TestNG provides out-of-the-box support for >>>> excluding specific test methods as well as groups of methods (where the >>>> groups are declared in source file annotations or Javadoc comments). >>>> Even better, it can do this on existing JUnit test code provided that >>>> the necessary meta-data (annotations if compiling to a 1.5 target; >>>> Javadoc comments if targeting 1.4 like we currently are). There is a >>>> utility available in the TestNG download and also in the Eclipse >>>> support >>>> plug-in that helps migrate directories of existing JUnit tests to >>>> TestNG >>>> by adding in the basic meta-data (although for me the Eclipse version >>>> also tried to break the test class inheritance from >>>> junit.framework.TestCase which was definitely not what was required). >>>> >>>> Perhaps ... just perhaps ... we should be looking at something like >>>> TestNG (or my wonderful "exclusions list" :-) ) to provide the >>>> granularity of test configuration that we need. >>>> >>>> Just a thought. >>>> >>> How 'bout that ;) >>> >>> geir >>> >>> >>>> Best regards, >>>> George >>>> >>>> [1] http://issues.apache.org/jira/browse/HARMONY-263 >>>> [2] http://testng.org >>>> >>>> >>>> >>>> Geir Magnusson Jr wrote: >>>> >>>>> Alexei Zakharov wrote: >>>>> >>>>> >>>>>> Hi, >>>>>> +1 for (3), but I think it will be better to define suite() method >>>>>> and >>>>>> enumerate passing tests there rather than to comment out the code. >>>>>> >>>>>> >>>>> I'm reluctant to see more dependencies on JUnit when we could control >>>>> >>> at >>> >>>>> a level higher in the build system. >>>>> >>>>> Hard to explain, I guess, but if our exclusions are buried in .java, I >>>>> would think that reporting and tracking over time is going to be much >>>>> harder. >>>>> >>>>> geir >>>>> >>>>> >>>>> >>>>>> 2006/6/27, Richard Liang <[EMAIL PROTECTED]>: >>>>>> >>>>>> >>>>>>> Hello Vladimir, >>>>>>> >>>>>>> +1 to option 3) . We shall comment the failed test cases out and add >>>>>>> FIXME to remind us to diagnose the problems later. ;-) >>>>>>> >>>>>>> Vladimir Ivanov wrote: >>>>>>> >>>>>>> >>>>>>>> I see your point. >>>>>>>> But I feel that we can miss regression in non-tested code if we >>>>>>>> exclude >>>>>>>> TestCases. >>>>>>>> Now, for example we miss testing of >>>>>>>> >>>>>>>> >>>>>>> java.lang.Class/Process/Thread/String >>>>>>> >>>>>>> >>>>>>>> and some other classes. >>>>>>>> >>>>>>>> While we have failing tests and don't want to pay attention to >>>>>>>> these >>>>>>>> failures we can: >>>>>>>> 1) Leave things as is - do not run TestCases with failing tests. >>>>>>>> 2) Split passing/failing TestCase into separate "failing TestCase" >>>>>>>> >>> and >>> >>>>>>>> "passing TestCase" and exclude "failing TestCases". When test or >>>>>>>> implementation is fixed we move tests from failing TestCase to >>>>>>>> >>> passing >>> >>>>>>>> TestCase. >>>>>>>> 3) Comment failing tests in TestCases. It is better to run 58 tests >>>>>>>> instead >>>>>>>> of 0 for String. >>>>>>>> 4) Run all TestCases, then, compare test run results with the 'list >>>>>>>> >>> of >>> >>>>>>>> known >>>>>>>> failures' and see whether new failures appeared. This, I think, is >>>>>>>> >>>>>>>> >>>>>>> better >>>>>>> >>>>>>> >>>>>>>> then 1, 2 and 3, but, overhead is that we support 2 lists - list of >>>>>>>> >>>>>>>> >>>>>>> known >>>>>>> >>>>>>> >>>>>>>> failing tests and exclude list where we put crashing tests. >>>>>>>> >>>>>>>> Thanks, Vladimir >>>>>>>> On 6/26/06, Tim Ellison <[EMAIL PROTECTED]> wrote: >>>>>>>> >>>>>>>> >>>>>>>>> Mikhail Loenko wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>>> Hi Vladimir, >>>>>>>>>> >>>>>>>>>> IMHO the tests are to verify that an update does not introduce >>>>>>>>>> any >>>>>>>>>> regression. So there are two options: remember which exactly >>>>>>>>>> >>>>>>>>>> >>>>>>> tests may >>>>>>> >>>>>>> >>>>>>>>> fail >>>>>>>>> >>>>>>>>> >>>>>>>>>> and remember that all tests must pass. I believe the latter >>>>>>>>>> one is >>>>>>>>>> >>>>>>>>>> >>>>>>>>> a bit >>>>>>>>> >>>>>>>>> >>>>>>>>>> easier and safer. >>>>>>>>>> >>>>>>>>>> >>>>>>>>> +1 >>>>>>>>> >>>>>>>>> Tim >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Mikhail >>>>>>>>>> >>>>>>>>>> 2006/6/26, Vladimir Ivanov <[EMAIL PROTECTED]>: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> Working with tests I noticed that we are excluding some tests >>>>>>>>>>> >>> just >>> >>>>>>>>>>> because >>>>>>>>>>> several tests from single TestCase fail. >>>>>>>>>>> >>>>>>>>>>> For example, the TestCase 'tests.api.java.lang.StringTest' >>>>>>>>>>> has 60 >>>>>>>>>>> tests and >>>>>>>>>>> only 2 of them fails. But the build excludes the whole TestCase >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> and we >>>>>>>>> >>>>>>>>> >>>>>>>>>>> just >>>>>>>>>>> miss testing of java.lang.String implementation. >>>>>>>>>>> >>>>>>>>>>> Do we really need to exclude TestCases in 'ant test' target? >>>>>>>>>>> >>>>>>>>>>> My suggestion is: do not exclude any tests until it crashes VM. >>>>>>>>>>> If somebody needs a list of tests that always passed a separated >>>>>>>>>>> target can >>>>>>>>>>> be added to build. >>>>>>>>>>> >>>>>>>>>>> Do you think we should add target 'test-all' to the build? >>>>>>>>>>> Thanks, Vladimir >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> Tim Ellison ([EMAIL PROTECTED]) >>>>>>>>> IBM Java technology centre, UK. >>>>>>>>> >>>>>>>>> >>>>>>> -- >>>>>>> Richard Liang >>>>>>> China Software Development Lab, IBM >>>>>>> >> >> >> >> --------------------------------------------------------------------- >> Terms of use : http://incubator.apache.org/harmony/mailing.html >> To unsubscribe, e-mail: [EMAIL PROTECTED] >> For additional commands, e-mail: [EMAIL PROTECTED] >> >> >> > > > --------------------------------------------------------------------- > Terms of use : http://incubator.apache.org/harmony/mailing.html > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > > --------------------------------------------------------------------- Terms of use : http://incubator.apache.org/harmony/mailing.html To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
