I see your point. But I feel that we can miss regression in non-tested code if we exclude TestCases. Now, for example we miss testing of java.lang.Class/Process/Thread/String and some other classes.
While we have failing tests and don't want to pay attention to these failures we can: 1) Leave things as is – do not run TestCases with failing tests. 2) Split passing/failing TestCase into separate "failing TestCase" and "passing TestCase" and exclude "failing TestCases". When test or implementation is fixed we move tests from failing TestCase to passing TestCase. 3) Comment failing tests in TestCases. It is better to run 58 tests instead of 0 for String. 4) Run all TestCases, then, compare test run results with the 'list of known failures' and see whether new failures appeared. This, I think, is better then 1, 2 and 3, but, overhead is that we support 2 lists - list of known failing tests and exclude list where we put crashing tests. Thanks, Vladimir On 6/26/06, Tim Ellison <[EMAIL PROTECTED]> wrote:
Mikhail Loenko wrote: > Hi Vladimir, > > IMHO the tests are to verify that an update does not introduce any > regression. So there are two options: remember which exactly tests may fail > and remember that all tests must pass. I believe the latter one is a bit > easier and safer. +1 Tim > Thanks, > Mikhail > > 2006/6/26, Vladimir Ivanov <[EMAIL PROTECTED]>: >> Hi, >> Working with tests I noticed that we are excluding some tests just >> because >> several tests from single TestCase fail. >> >> For example, the TestCase 'tests.api.java.lang.StringTest' has 60 >> tests and >> only 2 of them fails. But the build excludes the whole TestCase and we >> just >> miss testing of java.lang.String implementation. >> >> Do we really need to exclude TestCases in 'ant test' target? >> >> My suggestion is: do not exclude any tests until it crashes VM. >> If somebody needs a list of tests that always passed a separated >> target can >> be added to build. >> >> Do you think we should add target 'test-all' to the build? >> Thanks, Vladimir >> >> > > --------------------------------------------------------------------- > Terms of use : http://incubator.apache.org/harmony/mailing.html > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > -- Tim Ellison ([EMAIL PROTECTED]) IBM Java technology centre, UK. --------------------------------------------------------------------- Terms of use : http://incubator.apache.org/harmony/mailing.html To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
