Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/378/
7 tests failed. FAILED: org.apache.lucene.search.TestSearcherManager.testConcurrentIndexCloseSearchAndRefresh Error Message: Captured an uncaught exception in thread: Thread[id=1120, name=Thread-362, state=RUNNABLE, group=TGRP-TestSearcherManager] Stack Trace: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1120, name=Thread-362, state=RUNNABLE, group=TGRP-TestSearcherManager] Caused by: java.lang.RuntimeException: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/build/core/test/J0/temp/lucene.search.TestSearcherManager_80B654E3FFB0E8F-001/tempDir-001/_f8_FST50_0.tfp: Too many open files at __randomizedtesting.SeedInfo.seed([80B654E3FFB0E8F]:0) at org.apache.lucene.search.TestSearcherManager$10.run(TestSearcherManager.java:647) Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/build/core/test/J0/temp/lucene.search.TestSearcherManager_80B654E3FFB0E8F-001/tempDir-001/_f8_FST50_0.tfp: Too many open files at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48) at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81) at org.apache.lucene.mockfile.HandleTrackingFS.newByteChannel(HandleTrackingFS.java:271) at org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212) at java.nio.file.Files.newByteChannel(Files.java:361) at java.nio.file.Files.newByteChannel(Files.java:407) at org.apache.lucene.store.SimpleFSDirectory.openInput(SimpleFSDirectory.java:77) at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100) at org.apache.lucene.codecs.memory.FSTTermsReader.<init>(FSTTermsReader.java:79) at org.apache.lucene.codecs.memory.FSTPostingsFormat.fieldsProducer(FSTPostingsFormat.java:69) at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:292) at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:372) at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:113) at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:83) at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:172) at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:214) at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:106) at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:524) at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:294) at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:269) at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:259) at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156) at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58) at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) at org.apache.lucene.search.TestSearcherManager$10.run(TestSearcherManager.java:635) FAILED: junit.framework.TestSuite.org.apache.lucene.search.TestSearcherManager Error Message: The test or suite printed 29610 bytes to stdout and stderr, even though the limit was set to 8192 bytes. Increase the limit with @Limit, ignore it completely with @SuppressSysoutChecks or run with -Dtests.verbose=true Stack Trace: java.lang.AssertionError: The test or suite printed 29610 bytes to stdout and stderr, even though the limit was set to 8192 bytes. Increase the limit with @Limit, ignore it completely with @SuppressSysoutChecks or run with -Dtests.verbose=true at __randomizedtesting.SeedInfo.seed([80B654E3FFB0E8F]:0) at org.apache.lucene.util.TestRuleLimitSysouts.afterIfSuccessful(TestRuleLimitSysouts.java:211) at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterIfSuccessful(TestRuleAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:37) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) at java.lang.Thread.run(Thread.java:748) FAILED: org.apache.lucene.document.TestLatLonLineShapeQueries.testRandomBig Error Message: Java heap space Stack Trace: java.lang.OutOfMemoryError: Java heap space at __randomizedtesting.SeedInfo.seed([27546858426F5EB3:A00315D7D3362233]:0) at java.util.Arrays.copyOf(Arrays.java:3332) at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448) at java.lang.StringBuilder.append(StringBuilder.java:136) at org.apache.lucene.store.MockIndexInputWrapper.toString(MockIndexInputWrapper.java:224) at java.lang.String.valueOf(String.java:2994) at java.lang.StringBuilder.append(StringBuilder.java:131) at org.apache.lucene.store.BufferedChecksumIndexInput.<init>(BufferedChecksumIndexInput.java:34) at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:157) at org.apache.lucene.util.bkd.BKDWriter.verifyChecksum(BKDWriter.java:1430) at org.apache.lucene.util.bkd.BKDWriter.build(BKDWriter.java:1854) at org.apache.lucene.util.bkd.BKDWriter.finish(BKDWriter.java:1025) at org.apache.lucene.index.RandomCodec$1$1.writeField(RandomCodec.java:142) at org.apache.lucene.codecs.PointsWriter.mergeOneField(PointsWriter.java:62) at org.apache.lucene.codecs.PointsWriter.merge(PointsWriter.java:191) at org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.merge(Lucene60PointsWriter.java:145) at org.apache.lucene.codecs.asserting.AssertingPointsFormat$AssertingPointsWriter.merge(AssertingPointsFormat.java:142) at org.apache.lucene.index.SegmentMerger.mergePoints(SegmentMerger.java:188) at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:137) at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4446) at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4068) at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40) at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2170) at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5103) at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1612) at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1228) at org.apache.lucene.document.BaseLatLonShapeTestCase.indexRandomShapes(BaseLatLonShapeTestCase.java:255) at org.apache.lucene.document.BaseLatLonShapeTestCase.verify(BaseLatLonShapeTestCase.java:231) at org.apache.lucene.document.BaseLatLonShapeTestCase.doTestRandom(BaseLatLonShapeTestCase.java:212) at org.apache.lucene.document.BaseLatLonShapeTestCase.testRandomBig(BaseLatLonShapeTestCase.java:188) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) FAILED: org.apache.solr.cloud.RestartWhileUpdatingTest.test Error Message: There are still nodes recoverying - waited for 320 seconds Stack Trace: java.lang.AssertionError: There are still nodes recoverying - waited for 320 seconds at __randomizedtesting.SeedInfo.seed([885CD18E938D0AF5:8EE543D71670D]:0) at org.junit.Assert.fail(Assert.java:93) at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:185) at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:920) at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1477) at org.apache.solr.cloud.RestartWhileUpdatingTest.test(RestartWhileUpdatingTest.java:145) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935) at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971) at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985) at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010) at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) at java.lang.Thread.run(Thread.java:748) FAILED: junit.framework.TestSuite.org.apache.solr.cloud.RestartWhileUpdatingTest Error Message: 7 threads leaked from SUITE scope at org.apache.solr.cloud.RestartWhileUpdatingTest: 1) Thread[id=22711, name=searcherExecutor-8410-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2) Thread[id=22935, name=searcherExecutor-8472-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 3) Thread[id=22748, name=searcherExecutor-8424-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 4) Thread[id=22669, name=searcherExecutor-8396-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 5) Thread[id=22900, name=searcherExecutor-8458-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 6) Thread[id=22793, name=searcherExecutor-8438-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 7) Thread[id=22970, name=searcherExecutor-8486-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Stack Trace: com.carrotsearch.randomizedtesting.ThreadLeakError: 7 threads leaked from SUITE scope at org.apache.solr.cloud.RestartWhileUpdatingTest: 1) Thread[id=22711, name=searcherExecutor-8410-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2) Thread[id=22935, name=searcherExecutor-8472-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 3) Thread[id=22748, name=searcherExecutor-8424-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 4) Thread[id=22669, name=searcherExecutor-8396-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 5) Thread[id=22900, name=searcherExecutor-8458-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 6) Thread[id=22793, name=searcherExecutor-8438-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 7) Thread[id=22970, name=searcherExecutor-8486-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) at __randomizedtesting.SeedInfo.seed([885CD18E938D0AF5]:0) FAILED: junit.framework.TestSuite.org.apache.solr.cloud.RestartWhileUpdatingTest Error Message: There are still zombie threads that couldn't be terminated: 1) Thread[id=22711, name=searcherExecutor-8410-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2) Thread[id=22935, name=searcherExecutor-8472-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 3) Thread[id=22748, name=searcherExecutor-8424-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 4) Thread[id=22669, name=searcherExecutor-8396-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 5) Thread[id=22900, name=searcherExecutor-8458-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 6) Thread[id=22793, name=searcherExecutor-8438-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 7) Thread[id=22970, name=searcherExecutor-8486-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Stack Trace: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated: 1) Thread[id=22711, name=searcherExecutor-8410-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2) Thread[id=22935, name=searcherExecutor-8472-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 3) Thread[id=22748, name=searcherExecutor-8424-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 4) Thread[id=22669, name=searcherExecutor-8396-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 5) Thread[id=22900, name=searcherExecutor-8458-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 6) Thread[id=22793, name=searcherExecutor-8438-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 7) Thread[id=22970, name=searcherExecutor-8486-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) at __randomizedtesting.SeedInfo.seed([885CD18E938D0AF5]:0) FAILED: org.apache.solr.client.solrj.impl.CloudSolrClientTest.testParallelUpdateQTime Error Message: Error from server at https://127.0.0.1:35669/solr/collection1_shard2_replica_n3: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 404 Can not find: /solr/collection1_shard2_replica_n3/update</title> </head> <body><h2>HTTP ERROR 404</h2> <p>Problem accessing /solr/collection1_shard2_replica_n3/update. Reason: <pre> Can not find: /solr/collection1_shard2_replica_n3/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/> </body> </html> Stack Trace: org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at https://127.0.0.1:35669/solr/collection1_shard2_replica_n3: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 404 Can not find: /solr/collection1_shard2_replica_n3/update</title> </head> <body><h2>HTTP ERROR 404</h2> <p>Problem accessing /solr/collection1_shard2_replica_n3/update. Reason: <pre> Can not find: /solr/collection1_shard2_replica_n3/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/> </body> </html> at __randomizedtesting.SeedInfo.seed([6880BE5425CA039D:8658C5C56BB5D60E]:0) at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:551) at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1016) at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884) at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949) at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949) at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949) at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949) at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949) at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194) at org.apache.solr.client.solrj.impl.CloudSolrClientTest.testParallelUpdateQTime(CloudSolrClientTest.java:146) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935) at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971) at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985) at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:35669/solr/collection1_shard2_replica_n3: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 404 Can not find: /solr/collection1_shard2_replica_n3/update</title> </head> <body><h2>HTTP ERROR 404</h2> <p>Problem accessing /solr/collection1_shard2_replica_n3/update. Reason: <pre> Can not find: /solr/collection1_shard2_replica_n3/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/> </body> </html> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244) at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483) at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413) at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:528) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more Build Log: [...truncated 673 lines...] [junit4] Suite: org.apache.lucene.search.TestSearcherManager [junit4] 2> nov 15, 2018 7:34:18 PM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException [junit4] 2> WARNING: Uncaught exception in thread: Thread[Thread-362,5,TGRP-TestSearcherManager] [junit4] 2> java.lang.RuntimeException: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/build/core/test/J0/temp/lucene.search.TestSearcherManager_80B654E3FFB0E8F-001/tempDir-001/_f8_FST50_0.tfp: Too many open files [junit4] 2> at __randomizedtesting.SeedInfo.seed([80B654E3FFB0E8F]:0) [junit4] 2> at org.apache.lucene.search.TestSearcherManager$10.run(TestSearcherManager.java:647) [junit4] 2> Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/build/core/test/J0/temp/lucene.search.TestSearcherManager_80B654E3FFB0E8F-001/tempDir-001/_f8_FST50_0.tfp: Too many open files [junit4] 2> at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48) [junit4] 2> at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81) [junit4] 2> at org.apache.lucene.mockfile.HandleTrackingFS.newByteChannel(HandleTrackingFS.java:271) [junit4] 2> at org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212) [junit4] 2> at java.nio.file.Files.newByteChannel(Files.java:361) [junit4] 2> at java.nio.file.Files.newByteChannel(Files.java:407) [junit4] 2> at org.apache.lucene.store.SimpleFSDirectory.openInput(SimpleFSDirectory.java:77) [junit4] 2> at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100) [junit4] 2> at org.apache.lucene.codecs.memory.FSTTermsReader.<init>(FSTTermsReader.java:79) [junit4] 2> at org.apache.lucene.codecs.memory.FSTPostingsFormat.fieldsProducer(FSTPostingsFormat.java:69) [junit4] 2> at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:292) [junit4] 2> at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:372) [junit4] 2> at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:113) [junit4] 2> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:83) [junit4] 2> at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:172) [junit4] 2> at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:214) [junit4] 2> at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:106) [junit4] 2> at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:524) [junit4] 2> at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:294) [junit4] 2> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:269) [junit4] 2> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:259) [junit4] 2> at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) [junit4] 2> at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156) [junit4] 2> at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58) [junit4] 2> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) [junit4] 2> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) [junit4] 2> at org.apache.lucene.search.TestSearcherManager$10.run(TestSearcherManager.java:635) [junit4] 2> [junit4] 2> nov 15, 2018 7:34:28 PM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException [junit4] 2> WARNING: Uncaught exception in thread: Thread[Lucene Merge Thread #20,5,TGRP-TestSearcherManager] [junit4] 2> org.apache.lucene.index.MergePolicy$MergeException: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/build/core/test/J0/temp/lucene.search.TestSearcherManager_80B654E3FFB0E8F-001/tempDir-001/_p2_FST50_0.tfp: Too many open files [junit4] 2> at __randomizedtesting.SeedInfo.seed([80B654E3FFB0E8F]:0) [junit4] 2> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:704) [junit4] 2> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:684) [junit4] 2> Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/build/core/test/J0/temp/lucene.search.TestSearcherManager_80B654E3FFB0E8F-001/tempDir-001/_p2_FST50_0.tfp: Too many open files [junit4] 2> at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48) [junit4] 2> at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81) [junit4] 2> at org.apache.lucene.mockfile.HandleTrackingFS.newByteChannel(HandleTrackingFS.java:271) [junit4] 2> at org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212) [junit4] 2> at java.nio.file.Files.newByteChannel(Files.java:361) [junit4] 2> at java.nio.file.Files.newByteChannel(Files.java:407) [junit4] 2> at org.apache.lucene.store.SimpleFSDirectory.openInput(SimpleFSDirectory.java:77) [junit4] 2> at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100) [junit4] 2> at org.apache.lucene.codecs.memory.FSTTermsReader.<init>(FSTTermsReader.java:79) [junit4] 2> at org.apache.lucene.codecs.memory.FSTPostingsFormat.fieldsProducer(FSTPostingsFormat.java:69) [junit4] 2> at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:292) [junit4] 2> at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:372) [junit4] 2> at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:113) [junit4] 2> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:83) [junit4] 2> at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:172) [junit4] 2> at org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:709) [junit4] 2> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4380) [junit4] 2> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4068) [junit4] 2> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625) [junit4] 2> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662) [junit4] 2> [junit4] 2> nov 15, 2018 7:34:28 PM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException [junit4] 2> WARNING: Uncaught exception in thread: Thread[Thread-363,5,TGRP-TestSearcherManager] [junit4] 2> java.lang.RuntimeException: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/build/core/test/J0/temp/lucene.search.TestSearcherManager_80B654E3FFB0E8F-001/tempDir-001/_ow_MockRandom_0.tbk: Too many open files [junit4] 2> at __randomizedtesting.SeedInfo.seed([80B654E3FFB0E8F]:0) [junit4] 2> at org.apache.lucene.search.TestSearcherManager$11.run(TestSearcherManager.java:677) [junit4] 2> Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/build/core/test/J0/temp/lucene.search.TestSearcherManager_80B654E3FFB0E8F-001/tempDir-001/_ow_MockRandom_0.tbk: Too many open files [junit4] 2> at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48) [junit4] 2> at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81) [junit4] 2> at org.apache.lucene.mockfile.HandleTrackingFS.newByteChannel(HandleTrackingFS.java:271) [junit4] 2> at org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212) [junit4] 2> at java.nio.file.Files.newByteChannel(Files.java:361) [junit4] 2> at java.nio.file.Files.newByteChannel(Files.java:407) [junit4] 2> at org.apache.lucene.store.SimpleFSDirectory.openInput(SimpleFSDirectory.java:77) [junit4] 2> at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100) [junit4] 2> at org.apache.lucene.codecs.memory.FSTOrdTermsReader.<init>(FSTOrdTermsReader.java:89) [junit4] 2> at org.apache.lucene.codecs.mockrandom.MockRandomPostingsFormat.fieldsProducer(MockRandomPostingsFormat.java:303) [junit4] 2> at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:292) [junit4] 2> at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:372) [junit4] 2> at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:113) [junit4] 2> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:83) [junit4] 2> at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:172) [junit4] 2> at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:214) [junit4] 2> at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:106) [junit4] 2> at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:524) [junit4] 2> at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:103) [junit4] 2> at org.apache.lucene.search.SearcherManager.<init>(SearcherManager.java:108) [junit4] 2> at org.apache.lucene.search.SearcherManager.<init>(SearcherManager.java:76) [junit4] 2> at org.apache.lucene.search.TestSearcherManager$11.run(TestSearcherManager.java:665) [junit4] 2> [junit4] 2> nov 15, 2018 7:34:28 PM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException [junit4] 2> WARNING: Uncaught exception in thread: Thread[Lucene Merge Thread #21,5,TGRP-TestSearcherManager] [junit4] 2> org.apache.lucene.index.MergePolicy$MergeException: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/build/core/test/J0/temp/lucene.search.TestSearcherManager_80B654E3FFB0E8F-001/tempDir-001/_oz_Lucene50_0.pay: Too many open files [junit4] 2> at __randomizedtesting.SeedInfo.seed([80B654E3FFB0E8F]:0) [junit4] 2> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:704) [junit4] 2> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:684) [junit4] 2> Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/build/core/test/J0/temp/lucene.search.TestSearcherManager_80B654E3FFB0E8F-001/tempDir-001/_oz_Lucene50_0.pay: Too many open files [junit4] 2> at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48) [junit4] 2> at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81) [junit4] 2> at org.apache.lucene.mockfile.HandleTrackingFS.newByteChannel(HandleTrackingFS.java:271) [junit4] 2> at org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212) [junit4] 2> at java.nio.file.Files.newByteChannel(Files.java:361) [junit4] 2> at java.nio.file.Files.newByteChannel(Files.java:407) [junit4] 2> at org.apache.lucene.store.SimpleFSDirectory.openInput(SimpleFSDirectory.java:77) [junit4] 2> at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100) [junit4] 2> at org.apache.lucene.codecs.lucene50.Lucene50PostingsReader.<init>(Lucene50PostingsReader.java:94) [junit4] 2> at org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat.fieldsProducer(Lucene50PostingsFormat.java:442) [junit4] 2> at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:292) [junit4] 2> at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:372) [junit4] 2> at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:113) [junit4] 2> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:83) [junit4] 2> at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:172) [junit4] 2> at org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:709) [junit4] 2> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4380) [junit4] 2> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4068) [junit4] 2> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625) [junit4] 2> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662) [junit4] 2> [junit4] 2> nov 15, 2018 7:34:28 PM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException [junit4] 2> WARNING: Uncaught exception in thread: Thread[Thread-360,5,TGRP-TestSearcherManager] [junit4] 2> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed [junit4] 2> at __randomizedtesting.SeedInfo.seed([80B654E3FFB0E8F]:0) [junit4] 2> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:679) [junit4] 2> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:693) [junit4] 2> at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1606) [junit4] 2> at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1228) [junit4] 2> at org.apache.lucene.search.TestSearcherManager$8.run(TestSearcherManager.java:574) [junit4] 2> Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/build/core/test/J0/temp/lucene.search.TestSearcherManager_80B654E3FFB0E8F-001/tempDir-001/_oz_Lucene50_0.pay: Too many open files [junit4] 2> at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48) [junit4] 2> at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81) [junit4] 2> at org.apache.lucene.mockfile.HandleTrackingFS.newByteChannel(HandleTrackingFS.java:271) [junit4] 2> at org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212) [junit4] 2> at java.nio.file.Files.newByteChannel(Files.java:361) [junit4] 2> at java.nio.file.Files.newByteChannel(Files.java:407) [junit4] 2> at org.apache.lucene.store.SimpleFSDirectory.openInput(SimpleFSDirectory.java:77) [junit4] 2> at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100) [junit4] 2> at org.apache.lucene.codecs.lucene50.Lucene50PostingsReader.<init>(Lucene50PostingsReader.java:94) [junit4] 2> at org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat.fieldsProducer(Lucene50PostingsFormat.java:442) [junit4] 2> at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:292) [junit4] 2> at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:372) [junit4] 2> at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:113) [junit4] 2> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:83) [junit4] 2> at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:172) [junit4] 2> at org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:709) [junit4] 2> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4380) [junit4] 2> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4068) [junit4] 2> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625) [junit4] 2> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662) [junit4] 2> [junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory. [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestSearcherManager -Dtests.method=testConcurrentIndexCloseSearchAndRefresh -Dtests.seed=80B654E3FFB0E8F -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt -Dtests.locale=sk-SK -Dtests.timezone=Europe/Budapest -Dtests.asserts=true -Dtests.file.encoding=UTF-8 [junit4] ERROR 25.4s J0 | TestSearcherManager.testConcurrentIndexCloseSearchAndRefresh <<< [junit4] > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1120, name=Thread-362, state=RUNNABLE, group=TGRP-TestSearcherManager] [junit4] > Caused by: java.lang.RuntimeException: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/build/core/test/J0/temp/lucene.search.TestSearcherManager_80B654E3FFB0E8F-001/tempDir-001/_f8_FST50_0.tfp: Too many open files [junit4] > at __randomizedtesting.SeedInfo.seed([80B654E3FFB0E8F]:0) [junit4] > at org.apache.lucene.search.TestSearcherManager$10.run(TestSearcherManager.java:647) [junit4] > Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/build/core/test/J0/temp/lucene.search.TestSearcherManager_80B654E3FFB0E8F-001/tempDir-001/_f8_FST50_0.tfp: Too many open files [junit4] > at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48) [junit4] > at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81) [junit4] > at org.apache.lucene.mockfile.HandleTrackingFS.newByteChannel(HandleTrackingFS.java:271) [junit4] > at org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212) [junit4] > at java.nio.file.Files.newByteChannel(Files.java:361) [junit4] > at java.nio.file.Files.newByteChannel(Files.java:407) [junit4] > at org.apache.lucene.store.SimpleFSDirectory.openInput(SimpleFSDirectory.java:77) [junit4] > at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100) [junit4] > at org.apache.lucene.codecs.memory.FSTTermsReader.<init>(FSTTermsReader.java:79) [junit4] > at org.apache.lucene.codecs.memory.FSTPostingsFormat.fieldsProducer(FSTPostingsFormat.java:69) [junit4] > at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:292) [junit4] > at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:372) [junit4] > at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:113) [junit4] > at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:83) [junit4] > at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:172) [junit4] > at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:214) [junit4] > at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:106) [junit4] > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:524) [junit4] > at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:294) [junit4] > at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:269) [junit4] > at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:259) [junit4] > at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) [junit4] > at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156) [junit4] > at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58) [junit4] > at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) [junit4] > at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) [junit4] > at org.apache.lucene.search.TestSearcherManager$10.run(TestSearcherManager.java:635)Throwable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1142, name=Lucene Merge Thread #20, state=RUNNABLE, group=TGRP-TestSearcherManager] [junit4] > Caused by: org.apache.lucene.index.MergePolicy$MergeException: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/build/core/test/J0/temp/lucene.search.TestSearcherManager_80B654E3FFB0E8F-001/tempDir-001/_p2_FST50_0.tfp: Too many open files [junit4] > at __randomizedtesting.SeedInfo.seed([80B654E3FFB0E8F]:0) [junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:704) [junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:684) [junit4] > Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/build/core/test/J0/temp/lucene.search.TestSearcherManager_80B654E3FFB0E8F-001/tempDir-001/_p2_FST50_0.tfp: Too many open files [junit4] > at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48) [junit4] > at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81) [junit4] > at org.apache.lucene.mockfile.HandleTrackingFS.newByteChannel(HandleTrackingFS.java:271) [junit4] > at org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212) [junit4] > at java.nio.file.Files.newByteChannel(Files.java:361) [junit4] > at java.nio.file.Files.newByteChannel(Files.java:407) [junit4] > at org.apache.lucene.store.SimpleFSDirectory.openInput(SimpleFSDirectory.java:77) [junit4] > at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100) [junit4] > at org.apache.lucene.codecs.memory.FSTTermsReader.<init>(FSTTermsReader.java:79) [junit4] > at org.apache.lucene.codecs.memory.FSTPostingsFormat.fieldsProducer(FSTPostingsFormat.java:69) [junit4] > at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:292) [junit4] > at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:372) [junit4] > at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:113) [junit4] > at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:83) [junit4] > at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:172) [junit4] > at org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:709) [junit4] > at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4380) [junit4] > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4068) [junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625) [junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)Throwable #3: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1121, name=Thread-363, state=RUNNABLE, group=TGRP-TestSearcherManager] [junit4] > Caused by: java.lang.RuntimeException: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/build/core/test/J0/temp/lucene.search.TestSearcherManager_80B654E3FFB0E8F-001/tempDir-001/_ow_MockRandom_0.tbk: Too many open files [junit4] > at __randomizedtesting.SeedInfo.seed([80B654E3FFB0E8F]:0) [junit4] > at org.apache.lucene.search.TestSearcherManager$11.run(TestSearcherManager.java:677) [junit4] > Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/build/core/test/J0/temp/lucene.search.TestSearcherManager_80B654E3FFB0E8F-001/tempDir-001/_ow_MockRandom_0.tbk: Too many open files [junit4] > at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48) [junit4] > at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81) [junit4] > at org.apache.lucene.mockfile.HandleTrackingFS.newByteChannel(HandleTrackingFS.java:271) [junit4] > at org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212) [junit4] > at java.nio.file.Files.newByteChannel(Files.java:361) [junit4] > at java.nio.file.Files.newByteChannel(Files.java:407) [junit4] > at org.apache.lucene.store.SimpleFSDirectory.openInput(SimpleFSDirectory.java:77) [junit4] > at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100) [junit4] > at org.apache.lucene.codecs.memory.FSTOrdTermsReader.<init>(FSTOrdTermsReader.java:89) [junit4] > at org.apache.lucene.codecs.mockrandom.MockRandomPostingsFormat.fieldsProducer(MockRandomPostingsFormat.java:303) [junit4] > at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:292) [junit4] > at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:372) [junit4] > at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:113) [junit4] > at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:83) [junit4] > at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:172) [junit4] > at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:214) [junit4] > at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:106) [junit4] > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:524) [junit4] > at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:103) [junit4] > at org.apache.lucene.search.SearcherManager.<init>(SearcherManager.java:108) [junit4] > at org.apache.lucene.search.SearcherManager.<init>(SearcherManager.java:76) [junit4] > at org.apache.lucene.search.TestSearcherManager$11.run(TestSearcherManager.java:665)Throwable #4: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1118, name=Thread-360, state=RUNNABLE, group=TGRP-TestSearcherManager] [junit4] > Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed [junit4] > at __randomizedtesting.SeedInfo.seed([80B654E3FFB0E8F]:0) [junit4] > at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:679) [junit4] > at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:693) [junit4] > at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1606) [junit4] > at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1228) [junit4] > at org.apache.lucene.search.TestSearcherManager$8.run(TestSearcherManager.java:574) [junit4] > Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/build/core/test/J0/temp/lucene.search.TestSearcherManager_80B654E3FFB0E8F-001/tempDir-001/_oz_Lucene50_0.pay: Too many open files [junit4] > at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48) [junit4] > at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81) [junit4] > at org.apache.lucene.mockfile.HandleTrackingFS.newByteChannel(HandleTrackingFS.java:271) [junit4] > at org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212) [junit4] > at java.nio.file.Files.newByteChannel(Files.java:361) [junit4] > at java.nio.file.Files.newByteChannel(Files.java:407) [junit4] > at org.apache.lucene.store.SimpleFSDirectory.openInput(SimpleFSDirectory.java:77) [junit4] > at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100) [junit4] > at org.apache.lucene.codecs.lucene50.Lucene50PostingsReader.<init>(Lucene50PostingsReader.java:94) [junit4] > at org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat.fieldsProducer(Lucene50PostingsFormat.java:442) [junit4] > at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:292) [junit4] > at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:372) [junit4] > at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:113) [junit4] > at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:83) [junit4] > at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:172) [junit4] > at org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:709) [junit4] > at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4380) [junit4] > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4068) [junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625) [junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)Throwable #5: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1143, name=Lucene Merge Thread #21, state=RUNNABLE, group=TGRP-TestSearcherManager] [junit4] > Caused by: org.apache.lucene.index.MergePolicy$MergeException: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/build/core/test/J0/temp/lucene.search.TestSearcherManager_80B654E3FFB0E8F-001/tempDir-001/_oz_Lucene50_0.pay: Too many open files [junit4] > at __randomizedtesting.SeedInfo.seed([80B654E3FFB0E8F]:0) [junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:704) [junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:684) [junit4] > Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/build/core/test/J0/temp/lucene.search.TestSearcherManager_80B654E3FFB0E8F-001/tempDir-001/_oz_Lucene50_0.pay: Too many open files [junit4] > at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48) [junit4] > at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81) [junit4] > at org.apache.lucene.mockfile.HandleTrackingFS.newByteChannel(HandleTrackingFS.java:271) [junit4] > at org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212) [junit4] > at java.nio.file.Files.newByteChannel(Files.java:361) [junit4] > at java.nio.file.Files.newByteChannel(Files.java:407) [junit4] > at org.apache.lucene.store.SimpleFSDirectory.openInput(SimpleFSDirectory.java:77) [junit4] > at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100) [junit4] > at org.apache.lucene.codecs.lucene50.Lucene50PostingsReader.<init>(Lucene50PostingsReader.java:94) [junit4] > at org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat.fieldsProducer(Lucene50PostingsFormat.java:442) [junit4] > at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:292) [junit4] > at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:372) [junit4] > at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:113) [junit4] > at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:83) [junit4] > at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:172) [junit4] > at org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:709) [junit4] > at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4380) [junit4] > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4068) [junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625) [junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662) [junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {date=Lucene50(blocksize=128), extra35=Lucene50(blocksize=128), extra13=Lucene50(blocksize=128), extra36=PostingsFormat(name=MockRandom), extra14=PostingsFormat(name=MockRandom), extra33=FSTOrd50, extra11=FSTOrd50, extra8=FSTOrd50, extra34=FST50, extra12=FST50, extra9=FST50, extra17=Lucene50(blocksize=128), extra39=Lucene50(blocksize=128), extra18=PostingsFormat(name=MockRandom), extra15=FSTOrd50, extra37=FSTOrd50, extra38=FST50, extra16=FST50, extra2=Lucene50(blocksize=128), extra3=PostingsFormat(name=MockRandom), titleTokenized=PostingsFormat(name=MockRandom), extra0=FSTOrd50, extra19=FSTOrd50, extra1=FST50, body=Lucene50(blocksize=128), title=FSTOrd50, extra6=Lucene50(blocksize=128), extra7=PostingsFormat(name=MockRandom), extra4=FSTOrd50, extra5=FST50, packID=FSTOrd50, extra31=Lucene50(blocksize=128), extra32=PostingsFormat(name=MockRandom), extra10=PostingsFormat(name=MockRandom), extra30=FST50, extra24=Lucene50(blocksize=128), extra25=PostingsFormat(name=MockRandom), extra22=FSTOrd50, extra23=FST50, extra28=Lucene50(blocksize=128), extra29=PostingsFormat(name=MockRandom), docid=FST50, extra26=FSTOrd50, extra27=FST50, extra20=Lucene50(blocksize=128), extra21=PostingsFormat(name=MockRandom)}, docValues:{date=DocValuesFormat(name=Asserting), extra13=DocValuesFormat(name=Lucene70), extra14=DocValuesFormat(name=Asserting), extra11=DocValuesFormat(name=Lucene70), extra8=DocValuesFormat(name=Lucene70), extra12=DocValuesFormat(name=Lucene70), extra9=DocValuesFormat(name=Lucene70), extra17=DocValuesFormat(name=Lucene70), extra18=DocValuesFormat(name=Lucene70), extra15=DocValuesFormat(name=Asserting), extra16=DocValuesFormat(name=Lucene70), extra2=DocValuesFormat(name=Asserting), extra3=DocValuesFormat(name=Lucene70), titleTokenized=DocValuesFormat(name=Asserting), extra0=DocValuesFormat(name=Lucene70), extra19=DocValuesFormat(name=Asserting), extra1=DocValuesFormat(name=Asserting), body=DocValuesFormat(name=Asserting), extra6=DocValuesFormat(name=Lucene70), extra7=DocValuesFormat(name=Asserting), extra4=DocValuesFormat(name=Lucene70), extra5=DocValuesFormat(name=Lucene70), extra10=DocValuesFormat(name=Lucene70), extra35=DocValuesFormat(name=Lucene70), extra36=DocValuesFormat(name=Lucene70), docid_int=DocValuesFormat(name=Asserting), extra33=DocValuesFormat(name=Asserting), extra34=DocValuesFormat(name=Lucene70), extra39=DocValuesFormat(name=Asserting), extra37=DocValuesFormat(name=Lucene70), extra38=DocValuesFormat(name=Asserting), title=DocValuesFormat(name=Asserting), packID=DocValuesFormat(name=Lucene70), docid_intDV=DocValuesFormat(name=Lucene70), extra31=DocValuesFormat(name=Asserting), extra32=DocValuesFormat(name=Lucene70), extra30=DocValuesFormat(name=Asserting), extra24=DocValuesFormat(name=Lucene70), extra25=DocValuesFormat(name=Asserting), extra22=DocValuesFormat(name=Lucene70), extra23=DocValuesFormat(name=Lucene70), extra28=DocValuesFormat(name=Lucene70), extra29=DocValuesFormat(name=Lucene70), docid=DocValuesFormat(name=Lucene70), extra26=DocValuesFormat(name=Asserting), extra27=DocValuesFormat(name=Lucene70), extra20=DocValuesFormat(name=Lucene70), extra21=DocValuesFormat(name=Lucene70), titleDV=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=480, maxMBSortInHeap=5.417140369485799, sim=RandomSimilarity(queryNorm=true): {extra35=DFR I(ne)Z(0.3), extra13=IB LL-L3(800.0), extra36=DFR I(n)1, extra14=DFR I(ne)2, extra33=DFR I(F)Z(0.3), extra11=DFR I(ne)BZ(0.3), extra8=DFR I(n)LZ(0.3), extra34=BM25(k1=1.2,b=0.75), extra12=DFR I(F)L1, extra9=IB LL-D3(800.0), extra17=DFR GZ(0.3), extra39=DFR I(ne)LZ(0.3), extra18=DFR I(F)Z(0.3), extra15=DFR I(n)2, extra37=IB SPL-D1, extra38=DFR GL2, extra16=DFR I(F)LZ(0.3), extra2=DFR I(ne)L2, extra3=IB LL-D2, titleTokenized=DFR GL2, extra0=DFR G3(800.0), extra19=BM25(k1=1.2,b=0.75), extra1=IB SPL-D2, body=DFR I(n)Z(0.3), extra6=DFR I(n)BZ(0.3), extra7=IB SPL-LZ(0.3), extra4=DFR GLZ(0.3), extra5=ClassicSimilarity, extra31=DFR I(F)LZ(0.3), extra32=DFR GZ(0.3), extra10=DFR I(ne)B2, extra30=DFR I(n)2, extra24=DFR I(ne)L1, extra25=DFR I(n)L2, extra22=DFR I(n)L1, extra23=DFR GB2, extra28=DFR I(n)B2, extra29=DFR GBZ(0.3), extra26=DFR I(ne)L3(800.0), extra27=IB SPL-D3(800.0), extra20=IB LL-L1, extra21=DFR I(F)B1}, locale=sk-SK, timezone=Europe/Budapest [junit4] 2> NOTE: Linux 4.4.0-137-generic amd64/Oracle Corporation 1.8.0_191 (64-bit)/cpus=4,threads=1,free=308646856,total=437780480 [junit4] 2> NOTE: All tests run in this JVM: [TestLRUQueryCache, TestNotDocIdSet, TestIndexWriterLockRelease, TestSpansEnum, TestCheckIndex, TestIndexWriterOnJRECrash, TestDoubleRange, TestStressIndexing, TestMergeSchedulerExternal, TestFieldType, TestOmitPositions, TestMultiPhraseQuery, TestDirectMonotonic, TestRecyclingIntBlockAllocator, TestCachingTokenFilter, TestPrefixQuery, TestFastCompressionMode, TestTieredMergePolicy, TestFilterSpans, TestTotalHitCountCollector, TestBinaryDocument, TestSearcherManager] [junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory. [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestSearcherManager -Dtests.seed=80B654E3FFB0E8F -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt -Dtests.locale=sk-SK -Dtests.timezone=Europe/Budapest -Dtests.asserts=true -Dtests.file.encoding=UTF-8 [junit4] ERROR 0.00s J0 | TestSearcherManager (suite) <<< [junit4] > Throwable #1: java.lang.AssertionError: The test or suite printed 29610 bytes to stdout and stderr, even though the limit was set to 8192 bytes. Increase the limit with @Limit, ignore it completely with @SuppressSysoutChecks or run with -Dtests.verbose=true [junit4] > at __randomizedtesting.SeedInfo.seed([80B654E3FFB0E8F]:0) [junit4] > at java.lang.Thread.run(Thread.java:748) [junit4] Completed [120/486 (1!)] on J0 in 399.04s, 11 tests, 1 failure, 1 error <<< FAILURES! [...truncated 9595 lines...] [junit4] Suite: org.apache.lucene.document.TestLatLonLineShapeQueries [junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory. [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestLatLonLineShapeQueries -Dtests.method=testRandomBig -Dtests.seed=27546858426F5EB3 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt -Dtests.locale=en-CA -Dtests.timezone=Pacific/Guam -Dtests.asserts=true -Dtests.file.encoding=US-ASCII [junit4] ERROR 193s J2 | TestLatLonLineShapeQueries.testRandomBig <<< [junit4] > Throwable #1: java.lang.OutOfMemoryError: Java heap space [junit4] > at __randomizedtesting.SeedInfo.seed([27546858426F5EB3:A00315D7D3362233]:0) [junit4] > at java.util.Arrays.copyOf(Arrays.java:3332) [junit4] > at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) [junit4] > at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448) [junit4] > at java.lang.StringBuilder.append(StringBuilder.java:136) [junit4] > at org.apache.lucene.store.MockIndexInputWrapper.toString(MockIndexInputWrapper.java:224) [junit4] > at java.lang.String.valueOf(String.java:2994) [junit4] > at java.lang.StringBuilder.append(StringBuilder.java:131) [junit4] > at org.apache.lucene.store.BufferedChecksumIndexInput.<init>(BufferedChecksumIndexInput.java:34) [junit4] > at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:157) [junit4] > at org.apache.lucene.util.bkd.BKDWriter.verifyChecksum(BKDWriter.java:1430) [junit4] > at org.apache.lucene.util.bkd.BKDWriter.build(BKDWriter.java:1854) [junit4] > at org.apache.lucene.util.bkd.BKDWriter.finish(BKDWriter.java:1025) [junit4] > at org.apache.lucene.index.RandomCodec$1$1.writeField(RandomCodec.java:142) [junit4] > at org.apache.lucene.codecs.PointsWriter.mergeOneField(PointsWriter.java:62) [junit4] > at org.apache.lucene.codecs.PointsWriter.merge(PointsWriter.java:191) [junit4] > at org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.merge(Lucene60PointsWriter.java:145) [junit4] > at org.apache.lucene.codecs.asserting.AssertingPointsFormat$AssertingPointsWriter.merge(AssertingPointsFormat.java:142) [junit4] > at org.apache.lucene.index.SegmentMerger.mergePoints(SegmentMerger.java:188) [junit4] > at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:137) [junit4] > at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4446) [junit4] > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4068) [junit4] > at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40) [junit4] > at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2170) [junit4] > at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5103) [junit4] > at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1612) [junit4] > at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1228) [junit4] > at org.apache.lucene.document.BaseLatLonShapeTestCase.indexRandomShapes(BaseLatLonShapeTestCase.java:255) [junit4] > at org.apache.lucene.document.BaseLatLonShapeTestCase.verify(BaseLatLonShapeTestCase.java:231) [junit4] > at org.apache.lucene.document.BaseLatLonShapeTestCase.doTestRandom(BaseLatLonShapeTestCase.java:212) [junit4] > at org.apache.lucene.document.BaseLatLonShapeTestCase.testRandomBig(BaseLatLonShapeTestCase.java:188) [junit4] 2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/build/sandbox/test/J2/temp/lucene.document.TestLatLonLineShapeQueries_27546858426F5EB3-001 [junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {id=BlockTreeOrds(blocksize=128)}, docValues:{shape=DocValuesFormat(name=Memory), id=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=2015, maxMBSortInHeap=5.031953098640612, sim=RandomSimilarity(queryNorm=false): {}, locale=en-CA, timezone=Pacific/Guam [junit4] 2> NOTE: Linux 4.4.0-137-generic amd64/Oracle Corporation 1.8.0_191 (64-bit)/cpus=4,threads=1,free=363687528,total=463470592 [junit4] 2> NOTE: All tests run in this JVM: [TestLatLonShape, TestLatLonLineShapeQueries] [junit4] Completed [5/22 (1!)] on J2 in 596.58s, 4 tests, 1 error <<< FAILURES! [...truncated 42 lines...] [junit4] JVM J2: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/build/sandbox/test/temp/junit4-J2-20181115_204645_6008280401339066165663.sysout [junit4] >>> JVM J2 emitted unexpected output (verbatim) ---- [junit4] java.lang.OutOfMemoryError: Java heap space [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/heapdumps/java_pid8704.hprof ... [junit4] Heap dump file created [430321452 bytes in 0.938 secs] [junit4] <<< JVM J2: EOF ---- [...truncated 4623 lines...] [junit4] Suite: org.apache.solr.cloud.RestartWhileUpdatingTest [junit4] 2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.RestartWhileUpdatingTest_885CD18E938D0AF5-001/init-core-data-001 [junit4] 2> 2432570 INFO (SUITE-RestartWhileUpdatingTest-seed#[885CD18E938D0AF5]-worker) [ ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /tn/nk [junit4] 2> 2432591 INFO (TEST-RestartWhileUpdatingTest.test-seed#[885CD18E938D0AF5]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER [junit4] 2> 2432591 INFO (Thread-5806) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0 [junit4] 2> 2432591 INFO (Thread-5806) [ ] o.a.s.c.ZkTestServer Starting server [junit4] 2> 2432597 ERROR (Thread-5806) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes [junit4] 2> 2432691 INFO (TEST-RestartWhileUpdatingTest.test-seed#[885CD18E938D0AF5]) [ ] o.a.s.c.ZkTestServer start zk server on port:45347 [junit4] 2> 2432702 INFO (zkConnectionManagerCallback-5371-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected [junit4] 2> 2432718 INFO (zkConnectionManagerCallback-5373-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected [junit4] 2> 2432724 INFO (T [...truncated too long message...] h2> [junit4] 1> <p>Problem accessing /solr/collection1_shard2_replica_n3/update. Reason: [junit4] 1> <pre> Can not find: /solr/collection1_shard2_replica_n3/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/> [junit4] 1> [junit4] 1> </body> [junit4] 1> </html> [junit4] 1> , retry? 5 [junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory. [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=CloudSolrClientTest -Dtests.method=testParallelUpdateQTime -Dtests.seed=6880BE5425CA039D -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt -Dtests.locale=es-MX -Dtests.timezone=VST -Dtests.asserts=true -Dtests.file.encoding=UTF-8 [junit4] ERROR 2.97s J2 | CloudSolrClientTest.testParallelUpdateQTime <<< [junit4] > Throwable #1: org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at https://127.0.0.1:35669/solr/collection1_shard2_replica_n3: Expected mime type application/octet-stream but got text/html. <html> [junit4] > <head> [junit4] > <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> [junit4] > <title>Error 404 Can not find: /solr/collection1_shard2_replica_n3/update</title> [junit4] > </head> [junit4] > <body><h2>HTTP ERROR 404</h2> [junit4] > <p>Problem accessing /solr/collection1_shard2_replica_n3/update. Reason: [junit4] > <pre> Can not find: /solr/collection1_shard2_replica_n3/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/> [junit4] > </body> [junit4] > </html> [junit4] > at __randomizedtesting.SeedInfo.seed([6880BE5425CA039D:8658C5C56BB5D60E]:0) [junit4] > at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:551) [junit4] > at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1016) [junit4] > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884) [junit4] > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949) [junit4] > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949) [junit4] > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949) [junit4] > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949) [junit4] > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949) [junit4] > at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817) [junit4] > at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194) [junit4] > at org.apache.solr.client.solrj.impl.CloudSolrClientTest.testParallelUpdateQTime(CloudSolrClientTest.java:146) [junit4] > at java.lang.Thread.run(Thread.java:748) [junit4] > Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:35669/solr/collection1_shard2_replica_n3: Expected mime type application/octet-stream but got text/html. <html> [junit4] > <head> [junit4] > <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> [junit4] > <title>Error 404 Can not find: /solr/collection1_shard2_replica_n3/update</title> [junit4] > </head> [junit4] > <body><h2>HTTP ERROR 404</h2> [junit4] > <p>Problem accessing /solr/collection1_shard2_replica_n3/update. Reason: [junit4] > <pre> Can not find: /solr/collection1_shard2_replica_n3/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/> [junit4] > </body> [junit4] > </html> [junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607) [junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255) [junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244) [junit4] > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483) [junit4] > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413) [junit4] > at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:528) [junit4] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) [junit4] > at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) [junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [junit4] > ... 1 more [junit4] 1> - [junit4] 1> replica:core_node2 rstate:active live:true [junit4] 1> no one is recoverying [junit4] 1> - [junit4] 1> replica:core_node2 rstate:active live:true [junit4] 1> replica:core_node4 rstate:down live:true [junit4] 1> - [junit4] 1> replica:core_node2 rstate:active live:true [junit4] 1> replica:core_node4 rstate:recovering live:true [junit4] 1> - [junit4] 1> replica:core_node2 rstate:active live:true [junit4] 1> replica:core_node4 rstate:active live:true [junit4] 1> no one is recoverying [junit4] 1> - [junit4] 1> replica:core_node4 rstate:active live:true [junit4] 1> no one is recoverying [junit4] 1> 22:48:11.765 [TEST-CloudSolrClientTest.testRetryUpdatesWhenClusterStateIsStale-seed#[6880BE5425CA039D]] ERROR org.apache.solr.client.solrj.impl.CloudSolrClient - Request to collection [stale_state_test_col] failed due to (404) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:45672/solr/stale_state_test_col_shard1_replica_n1: Expected mime type application/octet-stream but got text/html. <html> [junit4] 1> <head> [junit4] 1> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> [junit4] 1> <title>Error 404 Can not find: /solr/stale_state_test_col_shard1_replica_n1/update</title> [junit4] 1> </head> [junit4] 1> <body><h2>HTTP ERROR 404</h2> [junit4] 1> <p>Problem accessing /solr/stale_state_test_col_shard1_replica_n1/update. Reason: [junit4] 1> <pre> Can not find: /solr/stale_state_test_col_shard1_replica_n1/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/> [junit4] 1> [junit4] 1> </body> [junit4] 1> </html> [junit4] 1> , retry? 0 [junit4] 1> 22:48:25.501 [SUITE-CloudSolrClientTest-seed#[6880BE5425CA039D]-worker] ERROR org.apache.zookeeper.server.ZooKeeperServer - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes [junit4] 2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-solrj/test/J2/temp/solr.client.solrj.impl.CloudSolrClientTest_6880BE5425CA039D-001 [junit4] 2> Nov 15, 2018 10:48:25 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks [junit4] 2> WARNING: Will linger awaiting termination of 1 leaked thread(s). [junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {multiDefault=Lucene50(blocksize=128), a_t=Lucene50(blocksize=128), title_s=Lucene50(blocksize=128), id=PostingsFormat(name=MockRandom), text=BlockTreeOrds(blocksize=128)}, docValues:{_version_=DocValuesFormat(name=Memory), multiDefault=DocValuesFormat(name=Asserting), a_t=DocValuesFormat(name=Asserting), title_s=DocValuesFormat(name=Asserting), intDefault=DocValuesFormat(name=Memory), id=DocValuesFormat(name=Lucene70), text=DocValuesFormat(name=Lucene70), timestamp=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=1592, maxMBSortInHeap=5.1909672590288425, sim=RandomSimilarity(queryNorm=true): {}, locale=es-MX, timezone=VST [junit4] 2> NOTE: Linux 4.4.0-137-generic amd64/Oracle Corporation 1.8.0_191 (64-bit)/cpus=4,threads=1,free=230506200,total=452460544 [junit4] 2> NOTE: All tests run in this JVM: [LessThanEqualToEvaluatorTest, TangentEvaluatorTest, NormalizeEvaluatorTest, ExclusiveOrEvaluatorTest, TestHash, SolrParamTest, HttpSolrClientConPoolTest, CosineEvaluatorTest, DocumentAnalysisResponseTest, TestXMLEscaping, SineEvaluatorTest, TestPolicy, CloudSolrClientRetryTest, PowerEvaluatorTest, UuidEvaluatorTest, MergeIndexesEmbeddedTest, NamedListTest, GraphTest, JsonQueryRequestIntegrationTest, TestCloudSolrClientConnections, RegressionEvaluatorTest, TestCollectionStateWatchers, ArrayEvaluatorTest, TestLBHttpSolrClient, CloudSolrClientBuilderTest, DomainMapTest, SolrQueryTest, NoOpResponseParserTest, OperationsTest, CloudSolrClientTest] [junit4] Completed [140/172 (1!)] on J2 in 74.64s, 16 tests, 1 error <<< FAILURES! [...truncated 8036 lines...] BUILD FAILED /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/build.xml:651: The following error occurred while executing this line: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/build.xml:585: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created: * java_pid8704.hprof Total time: 270 minutes 18 seconds Build step 'Invoke Ant' marked build as failure Archiving artifacts Recording test results Email was triggered for: Failure - Any Sending email for trigger: Failure - Any
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org