I just did a fresh svn checkout, and ran "ant clean tests" from the cmdline. This gives me 3 failures and 13 errors: http://www.computerhok.nl/tmp/junit-noframes.html
The WikiEngineTest.testSpacedNames1() always fails (Linux versus Mac/Windows) Here's an overview of more tests : http://www.computerhok.nl/tmp/jspwiki-testresult.html regards, Harry 2009/10/27 Andrew Jaquith <[email protected]> > Sounds like we have a few issues here: > > 1) Guitests. I'll see what I can find. Probably something minor. I > know the "tests" target runs all test classes ending in "*Test" and > ignores "AllTests", while Eclipse (and probably guitests) just runs > the AllTests classes. It's likely that one or more of the AllTests > classes is failing to include, oh, about 34 tests. :) > > 2) Graceful LDAP fail (inside the tests themselves). Any ideas on how > to implement? The easy way would be to look for a localhost listener > on 4890 (where the OpenLDAP test fixture listens) and then not run the > tests if it isn't found. Should they FAIL or PASS in that case? It > sounds like passing is the right thing to do. > > 3) Differences in your test pass rate versus mine. Not sure why your > "ant tests" run would produce different results than mine. I did try > running mine with a completely new, checked-out branch. Because I > can't know what changes you might have in your local branch, could you > check out a clean copy and diff the tree versus yours? SOMETHING is > different. Also, I'd like to know what Harry and others are seeing. > Gents, any clues? > > I agree that all three methods should return the same number of test > cases, and pass/fail the same ways. I also agree that tests should be > self-contained. That was part of the rationale for the Ant script > tweaks I checked in recently. > > Eclipse, by the way, hasn't been reliable for me, for testing, for a > while. I tend to exhaust memory somewhere around JSPWikiMarkupParser. > But I haven't tried it in the last few months (i.e. before my massive > bug-hunting campaign). > > Andrew > > > > On Tue, Oct 27, 2009 at 3:25 AM, Janne Jalkanen > <[email protected]> wrote: > >> Interestingly, I applied your most recent checkins applied (and I have > >> small one patch to JSPWikiMarkupParserTest that I haven't checked in). > >> I am running 100% clean, with no errors. Total number of tests: 1024 > >> -- a nice round number. :) WikiEngineTest.testOldVersionVars has > >> been running fine for me for a while. > > > > There's no way it should've run, unless you have some old code/config > files > > lying around. Can you check out a previous version to a clean directory > and > > see if it runs? > > > >> As a control case, I also checked out a new built from trunk, and > >> simply typed 'ant tests'. I used a vanilla build with absolutely no > >> customizations, even to build.properties. It ran completely clean also > >> except for 1 JSPWikiMarkupParserTest test (because I haven't checked > >> in that fix), 1024 tests total. > > > > Running the AllTests from Eclipse or with "ant guitests" results in 990 > test > > cases. "ant tests" is the only one giving 1024 tests, and I get 12 > failures > > and 14 errors for it. LdapAuthorizerTest, LdapUserDatabaseTest and > > XMLUserDatabaseTest all fail with all tests. > > > > What I find odd is that guitests and tests targets should give the same > > results, since they both are run from build.xml. > > > >> The only other item causing the discrepancy would be if you don't have > >> a local LDAP server running for the LDAP tests. Those should cause, at > >> most, 14 failures or errors. I'll add in some code to build.xml to set > >> up the LDAP fixtures and/or disable the tests if the OpenLDAP > >> executable isn't available. > > > > I think it's probably a better idea to do the test directly in the tests > > itself. The JCR TCK throws a NonExecutableException when the test case > > cannot be executed (and this shows up as a passed test). > > > > I think it's important that all three methods give the same number of > test > > cases; if the number is not reliable, it's too easy to forget to run > certain > > tests. > > > > Also, I sometimes run all tests for a given package from within Eclipse. > I'd > > like the test cases to be self-contained enough so that I don't have to > > remember which tests are supposed to run under which conditions. > > > > /Janne > > >
