I think it would be good to have a way of running 1. All API tests 2. All API tests except non-bug differences (TCK-style)
on any VM. So where one method tests both common behaviour and non-bug differences it should be split in two so that we don't lose the tests for common behaviour when running the TCK-style suite. I don't really have a strong opinion about whether we then identify non-bug difference test cases by annotations or by moving to a different class or otherwise, but HARMONY-263 looks like a huge amount of XML and a lot of work to implement for the whole of Harmony compared to doing something like having one "NonBugDifferencesTest.java" class per module. I also think there's another advantage in clearly stating which tests cases are for non-bug differences, which is that if someone makes a change that causes one of these tests to fail it's easy to find the corresponding JIRA or mailing list discussion, rather than just assume the test is wrong (and vice-versa we may currently have some tests that are wrong that we're assuming are non-bug differences). 2008/10/10 Regis <[EMAIL PROTECTED]>: > > Mark Hindess wrote: >> >> In message <[EMAIL PROTECTED]>, >> "Alexey Petrenko" writes: >>> >>> 2008/9/30 Mark Hindess <[EMAIL PROTECTED]>: >>>> >>>> In order to fix this bug I had to fix a number of invalid API tests. >>>> I think it would be a good idea to: >>>> >>>> 1) Run the API tests against the RI >>>> >>>> 2) Create exclude lists - with references to the relevant JIRA - for >>>> non-bug differences so the tests can be regularly run on the RI and >>>> expected to pass cleanly >>>> >>>> 3) Fix the non-non-bug (!) differences. >>> >>> This job really needs to be done... >> >> I had a quick look at how much work this might but immediately hit an >> issue that I think is best discussed first. The luni test in: >> >> api/common/org/apache/harmony/luni/tests/java/io/FileTest.java >> >> has 52 asserts. One (on line 2153) fails on the RI (because of a fix in >> HARMONY-3207 for which no non-bug difference jira was created AFAIK). >> >> Does the exclude list need to exclude the entire test - which would seem > > FileTest could pass on Harmony, I always think the exclude list is a place > to hold the tests which just only break on Harmony. >> >> to be a waste of potentially useful tests? Or is there a better way >> with junit 4? Or do we just start splitting out tests into separate >> source files - like FileNonBugDifferenceTest.java - for reference in >> exclude lists? > > I agree it waste useful tests, and I don't think splitting out tests is the > best way. FileTest just a simple example, requirements may become complex, > we have to split out more test files and they may have overlaps, it make > these files hard to manage and maintain. I thinks we need a way to group and > control tests on method level. And I found[1] > [1] https://issues.apache.org/jira/browse/HARMONY-263 > which extend the junit to support to exclude tests in method level in XML, > it maybe be helpful. >> >> I know we've discussed this many times before (along with repeatedly >> discounting testng) but I'd like to resolve this once and for all so we >> can use the tests to their full potential. >> >> This is a concrete example. How should we resolve this? >> >> I should stress that I have no strong opinion about testng or junit, but >> I do have a strong opinion about the need to understand the differences >> between the behaviour of our code and the RI particularly given the >> continuing absence of a TCK. To me this means running as many tests as >> possible on the RI to confirm that the tests are valid and documenting >> (close to the code if possible) or fixing every case where our behaviour >> doesn't match the RI. >> >> Regards, >> Mark. >> >> >> > > Best Regards, > Regis. > -- Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
