Hello David and Audrius, first of all, thanks for your answers to my mauve-setup question(s). I tried the MakeTestClassList program, which seems to work fine. However, after spending another few hours trying to get the whole testsuite running, my worst fears are confirmed... To summarize:
* writing new testcases is easy * running individual testcases is easy (from the shell or eclipse) * running parts of the testsuite via an index file works. * BUT there is no easy way to run the whole testsuite. A developer first has to create a custom list of exclusion testcases, before running the remaining fraction of the testsuite... Despite David's nice program, command line parameters, and exclusion testcase lists, I did not manage to run the whole testsuite reliably against today's classpath CVS. After passing some tests, jamvm encounters a test that throws some unexpected exception, runs into a stack-overflow, or complains about an IllegalStateException in loadClass(). In all cases, gnu.testlet.runner.Mauve does not run to the end, and the user doesn't even get the summary of test results. This basically leads to the situation that a developer needs to run the testsuite, try to understand the last error, remove the problematic test question from the list of testcases, and re-run. I tried this from scratch without David's exclusion list, but stopped after the first 20 iterations of this (with no end in sight). I then proceeded to remove larger chunks of the tests, and finally got a test report - but this in turn means that many tests were never run at all. --- The current all-testcases-at-once model used by gnu.testlet.runner.Mauve is not robust enough to survive all tests, while the opposite strategy of running each test in a new VM via (Simple)TestHarness is extremely wasteful due to the VM initialization overhead. Perhaps we could switch to a two-level model, where an outer process watches the progress of the inner runner process, and restarts the inner process whenever this crashes or deadlocks? Also, a test that doesn't compile should be marked as FAIL instead of stopping the testsuite to run at all... - Norman PS: a matrix of today's mauve experiments: * mauve+jdk1.5.0 with 'make check': doesn't even compile * mauve+jdk1.4.2 with 'make check': works, but subset of tests only * mauve+jdk1.4.2 with 'ant': doesn't even compile * mauve+jdk1.5.0 with 'ant': doesn't even compile * mauve+classpath+jamvm+jikes: compiles, but crashes on lots of testcases _______________________________________________ Classpath mailing list [email protected] http://lists.gnu.org/mailman/listinfo/classpath
