In message <[EMAIL PROTECTED]>, "Alexey Petrenko" writes: > > 2008/9/30 Mark Hindess <[EMAIL PROTECTED]>: > > > > In order to fix this bug I had to fix a number of invalid API tests. > > I think it would be a good idea to: > > > > 1) Run the API tests against the RI > > > > 2) Create exclude lists - with references to the relevant JIRA - for > > non-bug differences so the tests can be regularly run on the RI and > > expected to pass cleanly > > > > 3) Fix the non-non-bug (!) differences. > > This job really needs to be done...
I had a quick look at how much work this might but immediately hit an issue that I think is best discussed first. The luni test in: api/common/org/apache/harmony/luni/tests/java/io/FileTest.java has 52 asserts. One (on line 2153) fails on the RI (because of a fix in HARMONY-3207 for which no non-bug difference jira was created AFAIK). Does the exclude list need to exclude the entire test - which would seem to be a waste of potentially useful tests? Or is there a better way with junit 4? Or do we just start splitting out tests into separate source files - like FileNonBugDifferenceTest.java - for reference in exclude lists? I know we've discussed this many times before (along with repeatedly discounting testng) but I'd like to resolve this once and for all so we can use the tests to their full potential. This is a concrete example. How should we resolve this? I should stress that I have no strong opinion about testng or junit, but I do have a strong opinion about the need to understand the differences between the behaviour of our code and the RI particularly given the continuing absence of a TCK. To me this means running as many tests as possible on the RI to confirm that the tests are valid and documenting (close to the code if possible) or fixing every case where our behaviour doesn't match the RI. Regards, Mark.
