Perhaps we should have an isolated TCK-like test set, which can be run without any special setup or build - just a JRE. Perhaps this would create some dual-maintenance, but I agree with Mark's sentiment and concern and think some amount of sacrifice may be necessary. A couple of benefits to this isolation would including eliminating the complex per-project API/impl tests and just make everything impl tests, failing tests would never be excluded, they are either correct or not, failures mean a Harmony bug.
-Nathan On Thu, Oct 9, 2008 at 9:55 AM, Mark Hindess <[EMAIL PROTECTED]> wrote: > > In message <[EMAIL PROTECTED]>, > "Alexey Petrenko" writes: >> >> 2008/9/30 Mark Hindess <[EMAIL PROTECTED]>: >> > >> > In order to fix this bug I had to fix a number of invalid API tests. >> > I think it would be a good idea to: >> > >> > 1) Run the API tests against the RI >> > >> > 2) Create exclude lists - with references to the relevant JIRA - for >> > non-bug differences so the tests can be regularly run on the RI and >> > expected to pass cleanly >> > >> > 3) Fix the non-non-bug (!) differences. >> >> This job really needs to be done... > > I had a quick look at how much work this might but immediately hit an > issue that I think is best discussed first. The luni test in: > > api/common/org/apache/harmony/luni/tests/java/io/FileTest.java > > has 52 asserts. One (on line 2153) fails on the RI (because of a fix in > HARMONY-3207 for which no non-bug difference jira was created AFAIK). > > Does the exclude list need to exclude the entire test - which would seem > to be a waste of potentially useful tests? Or is there a better way > with junit 4? Or do we just start splitting out tests into separate > source files - like FileNonBugDifferenceTest.java - for reference in > exclude lists? > > I know we've discussed this many times before (along with repeatedly > discounting testng) but I'd like to resolve this once and for all so we > can use the tests to their full potential. > > This is a concrete example. How should we resolve this? > > I should stress that I have no strong opinion about testng or junit, but > I do have a strong opinion about the need to understand the differences > between the behaviour of our code and the RI particularly given the > continuing absence of a TCK. To me this means running as many tests as > possible on the RI to confirm that the tests are valid and documenting > (close to the code if possible) or fixing every case where our behaviour > doesn't match the RI. > > Regards, > Mark. > > >
