Interestingly, I applied your most recent checkins applied (and I have small one patch to JSPWikiMarkupParserTest that I haven't checked in). I am running 100% clean, with no errors. Total number of tests: 1024 -- a nice round number. :) WikiEngineTest.testOldVersionVars has been running fine for me for a while.
There's no way it should've run, unless you have some old code/config files lying around. Can you check out a previous version to a clean directory and see if it runs?
As a control case, I also checked out a new built from trunk, and simply typed 'ant tests'. I used a vanilla build with absolutely no customizations, even to build.properties. It ran completely clean also except for 1 JSPWikiMarkupParserTest test (because I haven't checked in that fix), 1024 tests total.
Running the AllTests from Eclipse or with "ant guitests" results in 990 test cases. "ant tests" is the only one giving 1024 tests, and I get 12 failures and 14 errors for it. LdapAuthorizerTest, LdapUserDatabaseTest and XMLUserDatabaseTest all fail with all tests.
What I find odd is that guitests and tests targets should give the same results, since they both are run from build.xml.
The only other item causing the discrepancy would be if you don't have a local LDAP server running for the LDAP tests. Those should cause, at most, 14 failures or errors. I'll add in some code to build.xml to set up the LDAP fixtures and/or disable the tests if the OpenLDAP executable isn't available.
I think it's probably a better idea to do the test directly in the tests itself. The JCR TCK throws a NonExecutableException when the test case cannot be executed (and this shows up as a passed test).
I think it's important that all three methods give the same number of test cases; if the number is not reliable, it's too easy to forget to run certain tests.
Also, I sometimes run all tests for a given package from within Eclipse. I'd like the test cases to be self-contained enough so that I don't have to remember which tests are supposed to run under which conditions.
/Janne
