I fixed the backwards test and removed the assertion there. This was forgotten to be merged back. The reason why this test fails:
TestRAMDir creates 10 threads that start to add files to a RAMDir and adds content to these files. The RAMFile updates its own size and also updates the size of the enclosing RAMDir (using AtomicLong). The problem is, that all threads do this in parallel and maybe another thread added the contents to the file and have updated the parents AtomicLong but not yet its own size. As update local size and in RAMDir is no longer atomic for the another thread, the assertion fails. In previous versions of RamDir both updates were one atomic op, synced to the directory. For speed reasons this was removed, so writing to RAMFiles no longer locks the parent directory. ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen <http://www.thetaphi.de/> http://www.thetaphi.de eMail: u...@thetaphi.de From: Robert Muir [mailto:rcm...@gmail.com] Sent: Sunday, February 21, 2010 8:44 PM To: java-dev@lucene.apache.org Subject: Re: (LUCENE-1844) Speed up junit tests Mike removed this assertion in LUCENE-2095, so this only happens in the backwards tests. On Sun, Feb 21, 2010 at 2:26 PM, Uwe Schindler <u...@thetaphi.de> wrote: Another test-bug that now shows as a real test failure (and not only in stderr as before, thanks to LUCENE-2274). Happens quite often, will check logs on Hudson, how often this happens. The test failure on my solaris box occurred in backwards branch of trunk. [junit] Testsuite: org.apache.lucene.store.TestRAMDirectory [junit] Tests run: 4, Failures: 1, Errors: 0, Time elapsed: 0.259 sec [junit] [junit] ------------- Standard Error ----------------- [junit] The following exceptions were thrown by threads: [junit] *** Thread: Thread-16978 *** [junit] junit.framework.AssertionFailedError: expected:<84992> but was:<86016> [junit] at junit.framework.Assert.fail(Assert.java:47) [junit] at junit.framework.Assert.failNotEquals(Assert.java:277) [junit] at junit.framework.Assert.assertEquals(Assert.java:64) [junit] at junit.framework.Assert.assertEquals(Assert.java:130) [junit] at junit.framework.Assert.assertEquals(Assert.java:136) [junit] at org.apache.lucene.store.TestRAMDirectory$1.run(TestRAMDirectory.java:129) [junit] ------------- ---------------- --------------- [junit] Testcase: testRAMDirectorySize(org.apache.lucene.store.TestRAMDirectory): FAILED [junit] Some threads throwed uncaught exceptions! [junit] junit.framework.AssertionFailedError: Some threads throwed uncaught exceptions! [junit] at org.apache.lucene.util.LuceneTestCase.tearDown(LuceneTestCase.java:142) [junit] at org.apache.lucene.store.TestRAMDirectory.tearDown(TestRAMDirectory.java:160) [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:250) [junit] [junit] [junit] TEST org.apache.lucene.store.TestRAMDirectory FAILED ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de <http://www.thetaphi.de/> eMail: u...@thetaphi.de From: Robert Muir [mailto:rcm...@gmail.com] Sent: Sunday, February 21, 2010 10:53 AM To: java-dev@lucene.apache.org Subject: Re: (LUCENE-1844) Speed up junit tests here is what i was worried about, if we cannot fix, i can revert back to forking. This is not reproduceable all the time: [junit] Testcase: testParallelMultiSort(org.apache.lucene.search.TestSort): Caused an ERROR [junit] java.util.ConcurrentModificationException [junit] java.lang.RuntimeException: java.util.ConcurrentModificationException [junit] at org.apache.lucene.search.ParallelMultiSearcher.foreach(ParallelMultiSearcher.java:216) [junit] at org.apache.lucene.search.ParallelMultiSearcher.search(ParallelMultiSearcher.java:121) [junit] at org.apache.lucene.search.Searcher.search(Searcher.java:49) [junit] at org.apache.lucene.search.TestSort.assertMatches(TestSort.java:965) [junit] at org.apache.lucene.search.TestSort.runMultiSorts(TestSort.java:891) [junit] at org.apache.lucene.search.TestSort.testParallelMultiSort(TestSort.java:629) [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:208) [junit] Caused by: java.util.ConcurrentModificationException [junit] at java.util.WeakHashMap$HashIterator.nextEntry(WeakHashMap.java:762) [junit] at java.util.WeakHashMap$KeyIterator.next(WeakHashMap.java:795) [junit] at org.apache.lucene.search.FieldCacheImpl.getCacheEntries(FieldCacheImpl.java:75) [junit] at org.apache.lucene.util.FieldCacheSanityChecker.checkSanity(FieldCacheSanityChecker.java:72) [junit] at org.apache.lucene.search.FieldCacheImpl$Cache.printNewInsanity(FieldCacheImpl.java:205) [junit] at org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:194) [junit] at org.apache.lucene.search.FieldCacheImpl.getInts(FieldCacheImpl.java:357) [junit] at org.apache.lucene.search.FieldCacheImpl$IntCache.createValue(FieldCacheImpl.java:373) [junit] at org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:183) [junit] at org.apache.lucene.search.FieldCacheImpl.getInts(FieldCacheImpl.java:357) [junit] at org.apache.lucene.search.FieldComparator$IntComparator.setNextReader(FieldComparator.java:438) [junit] at org.apache.lucene.search.TopFieldCollector$OneComparatorNonScoringCollector.setNextReader(TopFieldCollector.java:95) [junit] at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:207) [junit] at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:197) [junit] at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:175) [junit] at org.apache.lucene.search.MultiSearcher$MultiSearcherCallableWithSort.call(MultiSearcher.java:420) [junit] at org.apache.lucene.search.MultiSearcher$MultiSearcherCallableWithSort.call(MultiSearcher.java:394) [junit] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [junit] at java.util.concurrent.FutureTask.run(FutureTask.java:138) [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [junit] at java.lang.Thread.run(Thread.java:619) [junit] [junit] [junit] TEST org.apache.lucene.search.TestSort FAILED On Sun, Feb 21, 2010 at 2:10 AM, Robert Muir <rcm...@gmail.com> wrote: ok, can do. if it turns out there is a problem, we can revert until some work has been done on the tests. On Sat, Feb 20, 2010 at 5:45 PM, Michael McCandless <luc...@mikemccandless.com> wrote: Currently the tests run 1 jvm per test suite (eg, TestIndexWriter has its own jvm), I believe, and we haven't seen test failures... so I think for the most part tests are not interfering with each other (messing up global state). It should be less likely that we see interactions across test suites (but obviously still possible). I think we should commit this and then if there are somehow problems we can address them, then? Mike On Sun, Feb 14, 2010 at 6:27 AM, Robert Muir <rcm...@gmail.com> wrote: > its not just statics, I think we should really look at ensuring files are > closed etc, or eventually there will be a problem! > > I guess in general the tradeoff is, it requires us to have better test code. > > On Sun, Feb 14, 2010 at 5:53 AM, Uwe Schindler <u...@thetaphi.de> wrote: >> >> At least we should check all core tests to not set any static defaults >> without try...finally! Are there any possibilities inside Eclipse/other-IDEs >> to check this? >> >> Uwe >> >> ----- >> Uwe Schindler >> H.-H.-Meier-Allee 63, D-28213 Bremen >> http://www.thetaphi.de >> eMail: u...@thetaphi.de >> >> > -----Original Message----- >> > From: Michael McCandless [mailto:luc...@mikemccandless.com] >> > Sent: Sunday, February 14, 2010 11:43 AM >> > To: java-dev@lucene.apache.org >> > Subject: Re: (LUCENE-1844) Speed up junit tests >> > >> > Wow -- this is MUCH faster! I think we should switch... >> > >> > It seems like we use a batchtest for all core tests, then for all >> > back-compat tests, then once per contrib package? Ie, so "ant >> > test-core" uses one jvm? >> > >> > I think we should simply fix any badly behaved tests (that don't >> > restore statics). It's impressive we already have no test failures >> > when we do this... I guess our tests are already cleaning things up >> > (though also probably not often changing global state, or, changing it >> > in a way that'd lead other tests to fail). >> > >> > Mike >> > >> > On Sat, Feb 13, 2010 at 5:23 PM, Robert Muir <rcm...@gmail.com> wrote: >> > > On Fri, Nov 27, 2009 at 1:27 PM, Michael McCandless >> > > <luc...@mikemccandless.com> wrote: >> > >> >> > >> Also one thing I'd love to try is NOT forking the JVM for each test >> > >> (fork="no" in the junit task). I wonder how much time that'd buy... >> > >> >> > > >> > > it shaves off a good deal of time on my machine. >> > > >> > > 'ant test-core': 4 minutes, 39 seconds -> 3 minutes, 3 seconds >> > > 'ant test': 11 minutes, 8 seconds -> 7 minutes, 13 seconds >> > > >> > > however, it makes me a little nervous because i'm not sure all the >> > tests >> > > cleanup nicely if they change statics and stuff. >> > > anyway, here's the trivial patch (you don't want fork=no, because it >> > turns >> > > off assertions) >> > > >> > > Index: common-build.xml >> > > =================================================================== >> > > --- common-build.xml (revision 909395) >> > > +++ common-build.xml (working copy) >> > > @@ -398,7 +398,7 @@ >> > > </condition> >> > > <mkdir dir="@{junit.output.dir}"/> >> > > <junit printsummary="off" haltonfailure="no" >> > maxmemory="512M" >> > > - errorProperty="tests.failed" >> > failureProperty="tests.failed"> >> > > + errorProperty="tests.failed" >> > failureProperty="tests.failed" >> > > forkmode="perBatch"> >> > > <classpath refid="@{junit.classpath}"/> >> > > <assertions> >> > > <enable package="org.apache.lucene"/> >> > > >> > > -- >> > > Robert Muir >> > > rcm...@gmail.com >> > > >> > >> > --------------------------------------------------------------------- >> > To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org >> > For additional commands, e-mail: java-dev-h...@lucene.apache.org >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org >> For additional commands, e-mail: java-dev-h...@lucene.apache.org >> > > > > -- > Robert Muir > rcm...@gmail.com > --------------------------------------------------------------------- To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org -- Robert Muir rcm...@gmail.com -- Robert Muir rcm...@gmail.com -- Robert Muir rcm...@gmail.com
