Tim Ellison wrote:
> There was a submission that enabled finer control of failing tests (even
> by platform etc.)
>
> I may be wrong but commenting out tests usually means that they never
> get fixed;
Yes, that was my concern as well.
> even putting them into exclude clauses in the ant script is
> too hidden for me -- I prefer to see the exclusions and failures made
> available very clearly.
Well, if we did refactor into
TestFOO.java and TestFOO_Failures.java
and have an explicit include, people would clearly see the problematic
test cases and might be motivated to contribute by fixing...
geir
>
> Regards,
> Tim
>
> Alexei Zakharov wrote:
>> Hi,
>> +1 for (3), but I think it will be better to define suite() method and
>> enumerate passing tests there rather than to comment out the code.
>>
>> 2006/6/27, Richard Liang <[EMAIL PROTECTED]>:
>>> Hello Vladimir,
>>>
>>> +1 to option 3) . We shall comment the failed test cases out and add
>>> FIXME to remind us to diagnose the problems later. ;-)
>>>
>>> Vladimir Ivanov wrote:
>>>> I see your point.
>>>> But I feel that we can miss regression in non-tested code if we exclude
>>>> TestCases.
>>>> Now, for example we miss testing of
>>> java.lang.Class/Process/Thread/String
>>>> and some other classes.
>>>>
>>>> While we have failing tests and don't want to pay attention to these
>>>> failures we can:
>>>> 1) Leave things as is – do not run TestCases with failing tests.
>>>> 2) Split passing/failing TestCase into separate "failing TestCase" and
>>>> "passing TestCase" and exclude "failing TestCases". When test or
>>>> implementation is fixed we move tests from failing TestCase to passing
>>>> TestCase.
>>>> 3) Comment failing tests in TestCases. It is better to run 58 tests
>>>> instead
>>>> of 0 for String.
>>>> 4) Run all TestCases, then, compare test run results with the 'list of
>>>> known
>>>> failures' and see whether new failures appeared. This, I think, is
>>> better
>>>> then 1, 2 and 3, but, overhead is that we support 2 lists - list of
>>> known
>>>> failing tests and exclude list where we put crashing tests.
>>>>
>>>> Thanks, Vladimir
>>>> On 6/26/06, Tim Ellison <[EMAIL PROTECTED]> wrote:
>>>>> Mikhail Loenko wrote:
>>>>>> Hi Vladimir,
>>>>>>
>>>>>> IMHO the tests are to verify that an update does not introduce any
>>>>>> regression. So there are two options: remember which exactly
>>> tests may
>>>>> fail
>>>>>> and remember that all tests must pass. I believe the latter one is
>>>>> a bit
>>>>>> easier and safer.
>>>>> +1
>>>>>
>>>>> Tim
>>>>>
>>>>>> Thanks,
>>>>>> Mikhail
>>>>>>
>>>>>> 2006/6/26, Vladimir Ivanov <[EMAIL PROTECTED]>:
>>>>>>> Hi,
>>>>>>> Working with tests I noticed that we are excluding some tests just
>>>>>>> because
>>>>>>> several tests from single TestCase fail.
>>>>>>>
>>>>>>> For example, the TestCase 'tests.api.java.lang.StringTest' has 60
>>>>>>> tests and
>>>>>>> only 2 of them fails. But the build excludes the whole TestCase
>>>>> and we
>>>>>>> just
>>>>>>> miss testing of java.lang.String implementation.
>>>>>>>
>>>>>>> Do we really need to exclude TestCases in 'ant test' target?
>>>>>>>
>>>>>>> My suggestion is: do not exclude any tests until it crashes VM.
>>>>>>> If somebody needs a list of tests that always passed a separated
>>>>>>> target can
>>>>>>> be added to build.
>>>>>>>
>>>>>>> Do you think we should add target 'test-all' to the build?
>>>>>>> Thanks, Vladimir
>>>>>>>
>>>>>>>
>>>>>>
>>> ---------------------------------------------------------------------
>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>> To unsubscribe, e-mail: [EMAIL PROTECTED]
>>>>>> For additional commands, e-mail:
>>> [EMAIL PROTECTED]
>>>>>>
>>>>> --
>>>>>
>>>>> Tim Ellison ([EMAIL PROTECTED])
>>>>> IBM Java technology centre, UK.
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>> To unsubscribe, e-mail: [EMAIL PROTECTED]
>>>>> For additional commands, e-mail: [EMAIL PROTECTED]
>>>>>
>>>>>
>>> --
>>> Richard Liang
>>> China Software Development Lab, IBM
>>
>>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
