Looks like the lucene Jenkins slave needs an increase in the open files limit.
Here’s what I added to /etc/sysctl.conf on my Debian box to do this: fs.file-max = 2097152 Uwe, you coordinating the recent puppetization of the lucene slave - should we open an INFRA JIRA to make this change? -- Steve www.lucidworks.com > On Sep 7, 2016, at 6:12 AM, Apache Jenkins Server <[email protected]> > wrote: > > Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1111/ > > 10 tests failed. > FAILED: org.apache.lucene.index.TestIndexWriterReader.testDuringAddDelete > > Error Message: > /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/J2/temp/lucene.index.TestIndexWriterReader_61A624ED3323A1DA-001/index-SimpleFSDirectory-001/_24x_Lucene50_1.tim: > Too many open files > > Stack Trace: > java.nio.file.FileSystemException: > /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/J2/temp/lucene.index.TestIndexWriterReader_61A624ED3323A1DA-001/index-SimpleFSDirectory-001/_24x_Lucene50_1.tim: > Too many open files > at > __randomizedtesting.SeedInfo.seed([61A624ED3323A1DA:FF37536A19E223DA]:0) > at > org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48) > at > org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81) > at > org.apache.lucene.mockfile.HandleTrackingFS.newByteChannel(HandleTrackingFS.java:271) > at > org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212) > at > org.apache.lucene.mockfile.HandleTrackingFS.newByteChannel(HandleTrackingFS.java:240) > at > org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212) > at java.nio.file.Files.newByteChannel(Files.java:361) > at java.nio.file.Files.newByteChannel(Files.java:407) > at > org.apache.lucene.store.SimpleFSDirectory.openInput(SimpleFSDirectory.java:77) > at > org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:99) > at > org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.<init>(BlockTreeTermsReader.java:153) > at > org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat.fieldsProducer(Lucene50PostingsFormat.java:445) > at > org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:261) > at > org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:341) > at > org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:106) > at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:74) > at > org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145) > at > org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:197) > at > org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:103) > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:460) > at > org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:291) > at > org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:266) > at > org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:256) > at > org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) > at > org.apache.lucene.index.TestIndexWriterReader.testDuringAddDelete(TestIndexWriterReader.java:870) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921) > at > org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) > at > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) > at > org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) > at > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) > at > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367) > at > com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809) > at > com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460) > at > com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827) > at > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) > at > com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) > at > com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) > at > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) > at > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) > at > org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367) > at java.lang.Thread.run(Thread.java:745) > > > FAILED: > junit.framework.TestSuite.org.apache.lucene.index.TestIndexWriterReader > > Error Message: > 5 threads leaked from SUITE scope at > org.apache.lucene.index.TestIndexWriterReader: 1) Thread[id=1799, > name=Thread-1549, state=BLOCKED, group=TGRP-TestIndexWriterReader] at > org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37) > at > org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052) > at > org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910) > at > org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731) > at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948) > at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939) > at > org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565) > at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307) > at > org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848) > 2) Thread[id=1801, name=Thread-1551, state=RUNNABLE, > group=TGRP-TestIndexWriterReader] at > org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.get(CompressingTermVectorsReader.java:417) > at > org.apache.lucene.codecs.asserting.AssertingTermVectorsFormat$AssertingTermVectorsReader.get(AssertingTermVectorsFormat.java:71) > at > org.apache.lucene.codecs.TermVectorsWriter.merge(TermVectorsWriter.java:223) > at > org.apache.lucene.index.SegmentMerger.mergeVectors(SegmentMerger.java:210) > at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:144) > at > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4287) > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3864) > at > org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40) > at > org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052) > at > org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910) > at > org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731) > at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948) > at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939) > at > org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565) > at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307) > at > org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848) > 3) Thread[id=1802, name=Thread-1552, state=BLOCKED, > group=TGRP-TestIndexWriterReader] at > org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37) > at > org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052) > at > org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910) > at > org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731) > at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948) > at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939) > at > org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565) > at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307) > at > org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848) > 4) Thread[id=1800, name=Thread-1550, state=BLOCKED, > group=TGRP-TestIndexWriterReader] at > org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37) > at > org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052) > at > org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910) > at > org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731) > at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948) > at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939) > at > org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565) > at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307) > at > org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848) > 5) Thread[id=1798, name=Thread-1548, state=BLOCKED, > group=TGRP-TestIndexWriterReader] at > org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37) > at > org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052) > at > org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910) > at > org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731) > at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948) > at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939) > at > org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565) > at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307) > at > org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848) > > Stack Trace: > com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from > SUITE scope at org.apache.lucene.index.TestIndexWriterReader: > 1) Thread[id=1799, name=Thread-1549, state=BLOCKED, > group=TGRP-TestIndexWriterReader] > at > org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37) > at > org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052) > at > org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910) > at > org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731) > at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948) > at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939) > at > org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565) > at > org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307) > at > org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848) > 2) Thread[id=1801, name=Thread-1551, state=RUNNABLE, > group=TGRP-TestIndexWriterReader] > at > org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.get(CompressingTermVectorsReader.java:417) > at > org.apache.lucene.codecs.asserting.AssertingTermVectorsFormat$AssertingTermVectorsReader.get(AssertingTermVectorsFormat.java:71) > at > org.apache.lucene.codecs.TermVectorsWriter.merge(TermVectorsWriter.java:223) > at > org.apache.lucene.index.SegmentMerger.mergeVectors(SegmentMerger.java:210) > at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:144) > at > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4287) > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3864) > at > org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40) > at > org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052) > at > org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910) > at > org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731) > at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948) > at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939) > at > org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565) > at > org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307) > at > org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848) > 3) Thread[id=1802, name=Thread-1552, state=BLOCKED, > group=TGRP-TestIndexWriterReader] > at > org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37) > at > org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052) > at > org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910) > at > org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731) > at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948) > at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939) > at > org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565) > at > org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307) > at > org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848) > 4) Thread[id=1800, name=Thread-1550, state=BLOCKED, > group=TGRP-TestIndexWriterReader] > at > org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37) > at > org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052) > at > org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910) > at > org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731) > at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948) > at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939) > at > org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565) > at > org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307) > at > org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848) > 5) Thread[id=1798, name=Thread-1548, state=BLOCKED, > group=TGRP-TestIndexWriterReader] > at > org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37) > at > org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052) > at > org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910) > at > org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731) > at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948) > at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939) > at > org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565) > at > org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307) > at > org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848) > at __randomizedtesting.SeedInfo.seed([61A624ED3323A1DA]:0) > > > FAILED: > junit.framework.TestSuite.org.apache.solr.cloud.CollectionsAPIDistributedZkTest > > Error Message: > 1 thread leaked from SUITE scope at > org.apache.solr.cloud.CollectionsAPIDistributedZkTest: 1) > Thread[id=125227, name=searcherExecutor-10822-thread-1, state=WAITING, > group=TGRP-CollectionsAPIDistributedZkTest] at > sun.misc.Unsafe.park(Native Method) at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > Stack Trace: > com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from > SUITE scope at org.apache.solr.cloud.CollectionsAPIDistributedZkTest: > 1) Thread[id=125227, name=searcherExecutor-10822-thread-1, state=WAITING, > group=TGRP-CollectionsAPIDistributedZkTest] > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > at __randomizedtesting.SeedInfo.seed([66DE89DEA79A3E32]:0) > > > FAILED: > junit.framework.TestSuite.org.apache.solr.cloud.CollectionsAPIDistributedZkTest > > Error Message: > There are still zombie threads that couldn't be terminated: 1) > Thread[id=125227, name=searcherExecutor-10822-thread-1, state=WAITING, > group=TGRP-CollectionsAPIDistributedZkTest] at > sun.misc.Unsafe.park(Native Method) at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > Stack Trace: > com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie > threads that couldn't be terminated: > 1) Thread[id=125227, name=searcherExecutor-10822-thread-1, state=WAITING, > group=TGRP-CollectionsAPIDistributedZkTest] > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > at __randomizedtesting.SeedInfo.seed([66DE89DEA79A3E32]:0) > > > FAILED: org.apache.solr.cloud.hdfs.StressHdfsTest.test > > Error Message: > Could not find collection:delete_data_dir > > Stack Trace: > java.lang.AssertionError: Could not find collection:delete_data_dir > at > __randomizedtesting.SeedInfo.seed([66DE89DEA79A3E32:EE8AB604096653CA]:0) > at org.junit.Assert.fail(Assert.java:93) > at org.junit.Assert.assertTrue(Assert.java:43) > at org.junit.Assert.assertNotNull(Assert.java:526) > at > org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:154) > at > org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:139) > at > org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:134) > at > org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:856) > at > org.apache.solr.cloud.hdfs.StressHdfsTest.createAndDeleteCollection(StressHdfsTest.java:159) > at > org.apache.solr.cloud.hdfs.StressHdfsTest.test(StressHdfsTest.java:103) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921) > at > org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985) > at > org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960) > at > com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) > at > org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) > at > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) > at > org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) > at > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) > at > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367) > at > com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809) > at > com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460) > at > com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) > at > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) > at > com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) > at > com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) > at > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) > at > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) > at > org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367) > at java.lang.Thread.run(Thread.java:745) > > > FAILED: org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test > > Error Message: > Timeout occured while waiting response from server at: > https://127.0.0.1:39500/vum > > Stack Trace: > org.apache.solr.client.solrj.SolrServerException: Timeout occured while > waiting response from server at: https://127.0.0.1:39500/vum > at > org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:619) > at > org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261) > at > org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250) > at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219) > at > org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:400) > at > org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:516) > at > org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:180) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921) > at > org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985) > at > org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960) > at > com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) > at > org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) > at > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) > at > org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) > at > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) > at > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367) > at > com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809) > at > com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460) > at > com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) > at > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) > at > com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) > at > com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) > at > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) > at > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) > at > org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.net.SocketTimeoutException: Read timed out > at java.net.SocketInputStream.socketRead0(Native Method) > at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) > at java.net.SocketInputStream.read(SocketInputStream.java:170) > at java.net.SocketInputStream.read(SocketInputStream.java:141) > at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) > at sun.security.ssl.InputRecord.read(InputRecord.java:503) > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) > at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930) > at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) > at > org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) > at > org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) > at > org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) > at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) > at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) > at > org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) > at > org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) > at > org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167) > at > org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) > at > org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) > at > org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271) > at > org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) > at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) > at > org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) > at > org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) > at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) > at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) > at > org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:513) > ... 47 more > > > FAILED: > org.apache.solr.handler.component.DistributedFacetPivotLongTailTest.test > > Error Message: > IOException occured when talking to server at: > https://127.0.0.1:46729/_/d/collection1 > > Stack Trace: > org.apache.solr.client.solrj.SolrServerException: IOException occured when > talking to server at: https://127.0.0.1:46729/_/d/collection1 > at > __randomizedtesting.SeedInfo.seed([66DE89DEA79A3E32:EE8AB604096653CA]:0) > at > org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:622) > at > org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261) > at > org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250) > at > org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149) > at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173) > at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138) > at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:152) > at > org.apache.solr.handler.component.DistributedFacetPivotLongTailTest.test(DistributedFacetPivotLongTailTest.java:71) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921) > at > org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985) > at > org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960) > at > com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) > at > org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) > at > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) > at > org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) > at > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) > at > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367) > at > com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809) > at > com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460) > at > com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) > at > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) > at > com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) > at > com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) > at > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) > at > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) > at > org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to > 127.0.0.1:46729 [/127.0.0.1] failed: Read timed out > at > org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:143) > at > org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) > at > org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) > at > org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) > at > org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) > at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) > at > org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) > at > org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) > at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) > at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) > at > org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:513) > ... 48 more > Caused by: java.net.SocketTimeoutException: Read timed out > at java.net.SocketInputStream.socketRead0(Native Method) > at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) > at java.net.SocketInputStream.read(SocketInputStream.java:170) > at java.net.SocketInputStream.read(SocketInputStream.java:141) > at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) > at sun.security.ssl.InputRecord.read(InputRecord.java:503) > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) > at > sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375) > at > sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403) > at > sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387) > at > org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394) > at > org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353) > at > org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134) > ... 58 more > > > FAILED: org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test > > Error Message: > Timeout occured while waiting response from server at: https://127.0.0.1:35113 > > Stack Trace: > org.apache.solr.client.solrj.SolrServerException: Timeout occured while > waiting response from server at: https://127.0.0.1:35113 > at > org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:619) > at > org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261) > at > org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250) > at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219) > at > org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:400) > at > org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:516) > at > org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:180) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921) > at > org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985) > at > org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960) > at > com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) > at > org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) > at > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) > at > org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) > at > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) > at > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367) > at > com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809) > at > com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460) > at > com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) > at > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) > at > com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) > at > com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) > at > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) > at > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) > at > org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.net.SocketTimeoutException: Read timed out > at java.net.SocketInputStream.socketRead0(Native Method) > at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) > at java.net.SocketInputStream.read(SocketInputStream.java:170) > at java.net.SocketInputStream.read(SocketInputStream.java:141) > at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) > at sun.security.ssl.InputRecord.read(InputRecord.java:503) > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) > at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930) > at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) > at > org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) > at > org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) > at > org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) > at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) > at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) > at > org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) > at > org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) > at > org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167) > at > org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) > at > org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) > at > org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271) > at > org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) > at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) > at > org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) > at > org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) > at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) > at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) > at > org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:513) > ... 47 more > > > FAILED: > junit.framework.TestSuite.org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest > > Error Message: > ObjectTracker found 10 object(s) that were not released!!! > [InternalHttpClient, InternalHttpClient, InternalHttpClient, > InternalHttpClient, InternalHttpClient, InternalHttpClient, > InternalHttpClient, InternalHttpClient, InternalHttpClient, > InternalHttpClient] > > Stack Trace: > java.lang.AssertionError: ObjectTracker found 10 object(s) that were not > released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, > InternalHttpClient, InternalHttpClient, InternalHttpClient, > InternalHttpClient, InternalHttpClient, InternalHttpClient, > InternalHttpClient] > at __randomizedtesting.SeedInfo.seed([66DE89DEA79A3E32]:0) > at org.junit.Assert.fail(Assert.java:93) > at org.junit.Assert.assertTrue(Assert.java:43) > at org.junit.Assert.assertNull(Assert.java:551) > at > org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257) > at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) > at > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) > at > com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) > at > com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) > at > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) > at > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) > at > org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367) > at java.lang.Thread.run(Thread.java:745) > > > FAILED: > junit.framework.TestSuite.org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest > > Error Message: > 10 threads leaked from SUITE scope at > org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest: 1) > Thread[id=28289, name=Connection evictor, state=TIMED_WAITING, > group=TGRP-ConcurrentDeleteAndCreateCollectionTest] at > java.lang.Thread.sleep(Native Method) at > org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) > at java.lang.Thread.run(Thread.java:745) 2) Thread[id=28294, > name=Connection evictor, state=TIMED_WAITING, > group=TGRP-ConcurrentDeleteAndCreateCollectionTest] at > java.lang.Thread.sleep(Native Method) at > org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) > at java.lang.Thread.run(Thread.java:745) 3) Thread[id=28264, > name=Connection evictor, state=TIMED_WAITING, > group=TGRP-ConcurrentDeleteAndCreateCollectionTest] at > java.lang.Thread.sleep(Native Method) at > org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) > at java.lang.Thread.run(Thread.java:745) 4) Thread[id=28259, > name=Connection evictor, state=TIMED_WAITING, > group=TGRP-ConcurrentDeleteAndCreateCollectionTest] at > java.lang.Thread.sleep(Native Method) at > org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) > at java.lang.Thread.run(Thread.java:745) 5) Thread[id=28269, > name=Connection evictor, state=TIMED_WAITING, > group=TGRP-ConcurrentDeleteAndCreateCollectionTest] at > java.lang.Thread.sleep(Native Method) at > org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) > at java.lang.Thread.run(Thread.java:745) 6) Thread[id=28304, > name=Connection evictor, state=TIMED_WAITING, > group=TGRP-ConcurrentDeleteAndCreateCollectionTest] at > java.lang.Thread.sleep(Native Method) at > org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) > at java.lang.Thread.run(Thread.java:745) 7) Thread[id=28284, > name=Connection evictor, state=TIMED_WAITING, > group=TGRP-ConcurrentDeleteAndCreateCollectionTest] at > java.lang.Thread.sleep(Native Method) at > org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) > at java.lang.Thread.run(Thread.java:745) 8) Thread[id=28299, > name=Connection evictor, state=TIMED_WAITING, > group=TGRP-ConcurrentDeleteAndCreateCollectionTest] at > java.lang.Thread.sleep(Native Method) at > org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) > at java.lang.Thread.run(Thread.java:745) 9) Thread[id=28274, > name=Connection evictor, state=TIMED_WAITING, > group=TGRP-ConcurrentDeleteAndCreateCollectionTest] at > java.lang.Thread.sleep(Native Method) at > org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) > at java.lang.Thread.run(Thread.java:745) 10) Thread[id=28279, > name=Connection evictor, state=TIMED_WAITING, > group=TGRP-ConcurrentDeleteAndCreateCollectionTest] at > java.lang.Thread.sleep(Native Method) at > org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) > at java.lang.Thread.run(Thread.java:745) > > Stack Trace: > com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from > SUITE scope at org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest: > 1) Thread[id=28289, name=Connection evictor, state=TIMED_WAITING, > group=TGRP-ConcurrentDeleteAndCreateCollectionTest] > at java.lang.Thread.sleep(Native Method) > at > org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) > at java.lang.Thread.run(Thread.java:745) > 2) Thread[id=28294, name=Connection evictor, state=TIMED_WAITING, > group=TGRP-ConcurrentDeleteAndCreateCollectionTest] > at java.lang.Thread.sleep(Native Method) > at > org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) > at java.lang.Thread.run(Thread.java:745) > 3) Thread[id=28264, name=Connection evictor, state=TIMED_WAITING, > group=TGRP-ConcurrentDeleteAndCreateCollectionTest] > at java.lang.Thread.sleep(Native Method) > at > org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) > at java.lang.Thread.run(Thread.java:745) > 4) Thread[id=28259, name=Connection evictor, state=TIMED_WAITING, > group=TGRP-ConcurrentDeleteAndCreateCollectionTest] > at java.lang.Thread.sleep(Native Method) > at > org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) > at java.lang.Thread.run(Thread.java:745) > 5) Thread[id=28269, name=Connection evictor, state=TIMED_WAITING, > group=TGRP-ConcurrentDeleteAndCreateCollectionTest] > at java.lang.Thread.sleep(Native Method) > at > org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) > at java.lang.Thread.run(Thread.java:745) > 6) Thread[id=28304, name=Connection evictor, state=TIMED_WAITING, > group=TGRP-ConcurrentDeleteAndCreateCollectionTest] > at java.lang.Thread.sleep(Native Method) > at > org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) > at java.lang.Thread.run(Thread.java:745) > 7) Thread[id=28284, name=Connection evictor, state=TIMED_WAITING, > group=TGRP-ConcurrentDeleteAndCreateCollectionTest] > at java.lang.Thread.sleep(Native Method) > at > org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) > at java.lang.Thread.run(Thread.java:745) > 8) Thread[id=28299, name=Connection evictor, state=TIMED_WAITING, > group=TGRP-ConcurrentDeleteAndCreateCollectionTest] > at java.lang.Thread.sleep(Native Method) > at > org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) > at java.lang.Thread.run(Thread.java:745) > 9) Thread[id=28274, name=Connection evictor, state=TIMED_WAITING, > group=TGRP-ConcurrentDeleteAndCreateCollectionTest] > at java.lang.Thread.sleep(Native Method) > at > org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) > at java.lang.Thread.run(Thread.java:745) > 10) Thread[id=28279, name=Connection evictor, state=TIMED_WAITING, > group=TGRP-ConcurrentDeleteAndCreateCollectionTest] > at java.lang.Thread.sleep(Native Method) > at > org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) > at java.lang.Thread.run(Thread.java:745) > at __randomizedtesting.SeedInfo.seed([66DE89DEA79A3E32]:0) > > > > > Build Log: > [...truncated 913 lines...] > [junit4] Suite: org.apache.lucene.index.TestIndexWriterReader > [junit4] 2> NOTE: download the large Jenkins line-docs file by running > 'ant get-jenkins-line-docs' in the lucene directory. > [junit4] 2> NOTE: reproduce with: ant test > -Dtestcase=TestIndexWriterReader -Dtests.method=testDuringAddDelete > -Dtests.seed=61A624ED3323A1DA -Dtests.multiplier=2 -Dtests.nightly=true > -Dtests.slow=true > -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt > -Dtests.locale=mk -Dtests.timezone=Europe/Skopje -Dtests.asserts=true > -Dtests.file.encoding=UTF-8 > [junit4] ERROR 142s J2 | TestIndexWriterReader.testDuringAddDelete <<< > [junit4] > Throwable #1: java.nio.file.FileSystemException: > /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/J2/temp/lucene.index.TestIndexWriterReader_61A624ED3323A1DA-001/index-SimpleFSDirectory-001/_24x_Lucene50_1.tim: > Too many open files > [junit4] > at > __randomizedtesting.SeedInfo.seed([61A624ED3323A1DA:FF37536A19E223DA]:0) > [junit4] > at > org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48) > [junit4] > at > org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81) > [junit4] > at > org.apache.lucene.mockfile.HandleTrackingFS.newByteChannel(HandleTrackingFS.java:271) > [junit4] > at > org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212) > [junit4] > at > org.apache.lucene.mockfile.HandleTrackingFS.newByteChannel(HandleTrackingFS.java:240) > [junit4] > at > org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212) > [junit4] > at java.nio.file.Files.newByteChannel(Files.java:361) > [junit4] > at java.nio.file.Files.newByteChannel(Files.java:407) > [junit4] > at > org.apache.lucene.store.SimpleFSDirectory.openInput(SimpleFSDirectory.java:77) > [junit4] > at > org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:99) > [junit4] > at > org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.<init>(BlockTreeTermsReader.java:153) > [junit4] > at > org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat.fieldsProducer(Lucene50PostingsFormat.java:445) > [junit4] > at > org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:261) > [junit4] > at > org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:341) > [junit4] > at > org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:106) > [junit4] > at > org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:74) > [junit4] > at > org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145) > [junit4] > at > org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:197) > [junit4] > at > org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:103) > [junit4] > at > org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:460) > [junit4] > at > org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:291) > [junit4] > at > org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:266) > [junit4] > at > org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:256) > [junit4] > at > org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) > [junit4] > at > org.apache.lucene.index.TestIndexWriterReader.testDuringAddDelete(TestIndexWriterReader.java:870) > [junit4] > at java.lang.Thread.run(Thread.java:745) > [junit4] 2> NOTE: leaving temporary files on disk at: > /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/J2/temp/lucene.index.TestIndexWriterReader_61A624ED3323A1DA-001 > [junit4] 2> Sep 07, 2016 4:42:35 AM > com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks > [junit4] 2> WARNING: Will linger awaiting termination of 5 leaked > thread(s). > [junit4] 2> Sep 07, 2016 4:42:55 AM > com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks > [junit4] 2> SEVERE: 5 threads leaked from SUITE scope at > org.apache.lucene.index.TestIndexWriterReader: > [junit4] 2> 1) Thread[id=1799, name=Thread-1549, state=BLOCKED, > group=TGRP-TestIndexWriterReader] > [junit4] 2> at > org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910) > [junit4] 2> at > org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307) > [junit4] 2> at > org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848) > [junit4] 2> 2) Thread[id=1801, name=Thread-1551, state=RUNNABLE, > group=TGRP-TestIndexWriterReader] > [junit4] 2> at > org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.get(CompressingTermVectorsReader.java:417) > [junit4] 2> at > org.apache.lucene.codecs.asserting.AssertingTermVectorsFormat$AssertingTermVectorsReader.get(AssertingTermVectorsFormat.java:71) > [junit4] 2> at > org.apache.lucene.codecs.TermVectorsWriter.merge(TermVectorsWriter.java:223) > [junit4] 2> at > org.apache.lucene.index.SegmentMerger.mergeVectors(SegmentMerger.java:210) > [junit4] 2> at > org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:144) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4287) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3864) > [junit4] 2> at > org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910) > [junit4] 2> at > org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307) > [junit4] 2> at > org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848) > [junit4] 2> 3) Thread[id=1802, name=Thread-1552, state=BLOCKED, > group=TGRP-TestIndexWriterReader] > [junit4] 2> at > org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910) > [junit4] 2> at > org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307) > [junit4] 2> at > org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848) > [junit4] 2> 4) Thread[id=1800, name=Thread-1550, state=BLOCKED, > group=TGRP-TestIndexWriterReader] > [junit4] 2> at > org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910) > [junit4] 2> at > org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307) > [junit4] 2> at > org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848) > [junit4] 2> 5) Thread[id=1798, name=Thread-1548, state=BLOCKED, > group=TGRP-TestIndexWriterReader] > [junit4] 2> at > org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910) > [junit4] 2> at > org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307) > [junit4] 2> at > org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848) > [junit4] 2> Sep 07, 2016 4:42:55 AM > com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll > [junit4] 2> INFO: Starting to interrupt leaked threads: > [junit4] 2> 1) Thread[id=1799, name=Thread-1549, state=BLOCKED, > group=TGRP-TestIndexWriterReader] > [junit4] 2> 2) Thread[id=1801, name=Thread-1551, state=RUNNABLE, > group=TGRP-TestIndexWriterReader] > [junit4] 2> 3) Thread[id=1802, name=Thread-1552, state=BLOCKED, > group=TGRP-TestIndexWriterReader] > [junit4] 2> 4) Thread[id=1800, name=Thread-1550, state=BLOCKED, > group=TGRP-TestIndexWriterReader] > [junit4] 2> 5) Thread[id=1798, name=Thread-1548, state=BLOCKED, > group=TGRP-TestIndexWriterReader] > [junit4] 2> Sep 07, 2016 4:42:55 AM > com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll > [junit4] 2> INFO: All leaked threads terminated. > [junit4] 2> NOTE: test params are: codec=Asserting(Lucene62): > {field1=Lucene50(blocksize=128), field=Lucene50(blocksize=128), > indexname=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), > foo=Lucene50(blocksize=128), > id=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), > > field3=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), > field2=FSTOrd50, field5=Lucene50(blocksize=128), > field4=Lucene50(blocksize=128)}, docValues:{}, maxPointsInLeafNode=1314, > maxMBSortInHeap=5.621346163809926, sim=RandomSimilarity(queryNorm=true): > {field1=DFR I(F)B1, field=DFR I(n)L2, indexname=DFR GLZ(0.3), foo=DFR > GLZ(0.3), field3=DFR I(ne)Z(0.3), field2=IB SPL-D1, field5=DFR I(F)1, > field4=DFR I(F)LZ(0.3)}, locale=mk, timezone=Europe/Skopje > [junit4] 2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation > 1.8.0_102 (64-bit)/cpus=4,threads=1,free=139897912,total=268959744 > [junit4] 2> NOTE: All tests run in this JVM: [Test2BPagedBytes, > TestStressIndexing, TestGeoUtils, TestNoMergePolicy, TestRollback, > TestBlockPostingsFormat, TestTermQuery, TestTopDocsMerge, TestFilterIterator, > TestStressDeletes, TestIntsRef, TestDocBoost, TestNamedSPILoader, > TestTransactions, TestRollingBuffer, TestFilterWeight, > TestIndexWriterThreadsToSegments, TestLogMergePolicy, > TestHighCompressionMode, TestDocValuesIndexing, TestRegexpRandom2, > TestSingleInstanceLockFactory, TestLucene62SegmentInfoFormat, > TestSegmentMerger, TestIndexWriterCommit, TestNRTCachingDirectory, > TestAllFilesHaveCodecHeader, TestFuzzyQuery, Test2BPostings, > TestDocumentsWriterDeleteQueue, TestByteBlockPool, TestScorerPerf, > TestPayloadsOnVectors, TestParallelLeafReader, TestSegmentTermDocs, > TestParallelTermEnum, TestDocument, TestVersion, TestAtomicUpdate, > TestConjunctionDISI, TestBagOfPostings, TestMultiTermConstantScore, > TestSnapshotDeletionPolicy, TestMmapDirectory, TestSpansEnum, > TestSpanNotQuery, TestTimSorter, TestWildcardRandom, TestInPlaceMergeSorter, > TestFixedBitDocIdSet, TestSimilarity, TestSearchWithThreads, > TestCharacterUtils, TestUpgradeIndexMergePolicy, TestBlendedTermQuery, > TestBooleanQuery, TestArrayUtil, TestSimpleSearchEquivalence, > TestSpanMultiTermQueryWrapper, TestDeletionPolicy, TestMultiPhraseEnum, > TestLazyProxSkipping, TestForUtil, TestSpanExplanations, > TestReaderWrapperDVTypeCheck, TestFastCompressionMode, TestRegexpQuery, > TestTopFieldCollector, TestMSBRadixSorter, TestIndexWriterMerging, > TestFileSwitchDirectory, TestIndexWriter, TestIndexWriterReader] > [junit4] 2> NOTE: download the large Jenkins line-docs file by running > 'ant get-jenkins-line-docs' in the lucene directory. > [junit4] 2> NOTE: reproduce with: ant test > -Dtestcase=TestIndexWriterReader -Dtests.seed=61A624ED3323A1DA > -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true > -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt > -Dtests.locale=mk -Dtests.timezone=Europe/Skopje -Dtests.asserts=true > -Dtests.file.encoding=UTF-8 > [junit4] ERROR 0.00s J2 | TestIndexWriterReader (suite) <<< > [junit4] > Throwable #1: > com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from > SUITE scope at org.apache.lucene.index.TestIndexWriterReader: > [junit4] > 1) Thread[id=1799, name=Thread-1549, state=BLOCKED, > group=TGRP-TestIndexWriterReader] > [junit4] > at > org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37) > [junit4] > at > org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052) > [junit4] > at > org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910) > [junit4] > at > org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731) > [junit4] > at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948) > [junit4] > at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939) > [junit4] > at > org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565) > [junit4] > at > org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307) > [junit4] > at > org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848) > [junit4] > 2) Thread[id=1801, name=Thread-1551, state=RUNNABLE, > group=TGRP-TestIndexWriterReader] > [junit4] > at > org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.get(CompressingTermVectorsReader.java:417) > [junit4] > at > org.apache.lucene.codecs.asserting.AssertingTermVectorsFormat$AssertingTermVectorsReader.get(AssertingTermVectorsFormat.java:71) > [junit4] > at > org.apache.lucene.codecs.TermVectorsWriter.merge(TermVectorsWriter.java:223) > [junit4] > at > org.apache.lucene.index.SegmentMerger.mergeVectors(SegmentMerger.java:210) > [junit4] > at > org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:144) > [junit4] > at > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4287) > [junit4] > at > org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3864) > [junit4] > at > org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40) > [junit4] > at > org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052) > [junit4] > at > org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910) > [junit4] > at > org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731) > [junit4] > at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948) > [junit4] > at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939) > [junit4] > at > org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565) > [junit4] > at > org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307) > [junit4] > at > org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848) > [junit4] > 3) Thread[id=1802, name=Thread-1552, state=BLOCKED, > group=TGRP-TestIndexWriterReader] > [junit4] > at > org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37) > [junit4] > at > org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052) > [junit4] > at > org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910) > [junit4] > at > org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731) > [junit4] > at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948) > [junit4] > at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939) > [junit4] > at > org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565) > [junit4] > at > org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307) > [junit4] > at > org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848) > [junit4] > 4) Thread[id=1800, name=Thread-1550, state=BLOCKED, > group=TGRP-TestIndexWriterReader] > [junit4] > at > org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37) > [junit4] > at > org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052) > [junit4] > at > org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910) > [junit4] > at > org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731) > [junit4] > at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948) > [junit4] > at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939) > [junit4] > at > org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565) > [junit4] > at > org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307) > [junit4] > at > org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848) > [junit4] > 5) Thread[id=1798, name=Thread-1548, state=BLOCKED, > group=TGRP-TestIndexWriterReader] > [junit4] > at > org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37) > [junit4] > at > org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052) > [junit4] > at > org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910) > [junit4] > at > org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731) > [junit4] > at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948) > [junit4] > at > org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939) > [junit4] > at > org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565) > [junit4] > at > org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307) > [junit4] > at > org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848) > [junit4] > at > __randomizedtesting.SeedInfo.seed([61A624ED3323A1DA]:0) > [junit4] Completed [200/432 (1!)] on J2 in 252.56s, 23 tests, 2 errors <<< > FAILURES! > > [...truncated 10658 lines...] > [junit4] Suite: org.apache.solr.cloud.hdfs.StressHdfsTest > [junit4] 2> Creating dataDir: > /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_66DE89DEA79A3E32-001/init-core-data-001 > [junit4] 2> 814899 INFO > (SUITE-StressHdfsTest-seed#[66DE89DEA79A3E32]-worker) [ ] > o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: > @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776) > [junit4] 2> 814899 INFO > (SUITE-StressHdfsTest-seed#[66DE89DEA79A3E32]-worker) [ ] > o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /_a/m > [junit4] 1> Formatting using clusterid: testClusterID > [junit4] 2> 815026 WARN > (SUITE-StressHdfsTest-seed#[66DE89DEA79A3E32]-worker) [ ] > o.a.h.m.i.MetricsConfig Cannot locate configuration: tried > hadoop-metrics2-namenode.properties,hadoop-metrics2.properties > [junit4] 2> 815069 WARN > (SUITE-StressHdfsTest-seed#[66DE89DEA79A3E32]-worker) [ ] > o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j > [junit4] 2> 815081 INFO > (SUITE-StressHdfsTest-seed#[66DE89DEA79A3E32]-worker) [ ] o.m.log > jetty-6.1.26 > [junit4] 2> 815125 INFO > (SUITE-StressHdfsTest-seed#[66DE89DEA79A3E32]-worker) [ ] o.m.log Extract > jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs > to ./temp/Jetty_localhost_33702_hdfs____13ay83/webapp > [junit4] 2> 816226 INFO > (SUITE-StressHdfsTest-seed#[66DE89DEA79A3E32]-worker) [ ] o.m.log Started > HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33702 > [junit4] 2> 817177 WARN > (SUITE-StressHdfsTest-seed#[66DE89DEA79A3E32]-worker) [ ] > o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j > [junit4] 2> 817179 INFO > (SUITE-StressHdfsTest-seed#[66DE89DEA79A3E32]-worker) [ ] o.m.log > jetty-6.1.26 > [junit4] 2> 817200 INFO > (SUITE-StressHdfsTest-seed#[66DE89DEA79A3E32]-worker) [ ] o.m.log Extract > jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode > to ./temp/Jetty_localhost_46024_datanode____.2mcfop/webapp > [junit4] 2> 818238 INFO > (SUITE-StressHdfsTest-seed#[66DE89DEA79A3E32]-worker) [ ] o.m.log Started > HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46024 > [junit4] 2> 818469 WARN > (SUITE-StressHdfsTest-seed#[66DE89DEA79A3E32]-worker) [ ] > o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j > [junit4] 2> 818470 INFO > (SUITE-StressHdfsTest-seed#[66DE89DEA79A3E32]-worker) [ ] o.m.log > jetty-6.1.26 > [junit4] 2> 818502 INFO > (SUITE-StressHdfsTest-seed#[66DE89DEA79A3E32]-worker) [ ] o.m.log Extract > jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode > to ./temp/Jetty_localhost_55265_datanode____.v9bv0o/webapp > [junit4] 2> 818789 INFO (IPC Server handler 6 on 59517) [ ] > BlockStateChange BLOCK* processReport: from storage > DS-124aedd7-53e1-4abe-9593-1749020619f8 node > DatanodeRegistration(127.0.0.1:39806, > datanodeUuid=44048ab0-2317-4266-b18a-947e4bc5cd29, infoPort=51796, > infoSecurePort=0, ipcPort=39160, > storageInfo=lv=-56;cid=testClusterID;nsid=2101821981;c=0), blocks: 0, > hasStaleStorage: true, processing time: 0 msecs > [junit4] 2> 818789 INFO (IPC Server handler 6 on 59517) [ ] > BlockStateChange BLOCK* processReport: from storage > DS-0427d4bd-1eeb-4af5-a408-0b312310f3bb node > DatanodeRegistration(127.0.0.1:39806, > datanodeUuid=44048ab0-2317-4266-b18a-947e4bc5cd29, infoPort=51796, > infoSecurePort=0, ipcPort=39160, > storageInfo=lv=-56;cid=testClusterID;nsid=2101821981;c=0), blocks: 0, > hasStaleStorage: false, processing time: 0 msecs > [junit4] 2> 819366 INFO > (SUITE-StressHdfsTest-seed#[66DE89DEA79A3E32]-worker) [ ] o.m.log Started > HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:55265 > [junit4] 2> 819712 INFO (IPC Server handler 3 on 59517) [ ] > BlockStateChange BLOCK* processReport: from storage > DS-0a413b87-a6df-47f1-b731-f3928bdb7bd0 node > DatanodeRegistration(127.0.0.1:38537, > datanodeUuid=6109fe6a-418b-4810-8965-110624cd978c, infoPort=48162, > infoSecurePort=0, ipcPort=33212, > storageInfo=lv=-56;cid=testClusterID;nsid=2101821981;c=0), blocks: 0, > hasStaleStorage: true, processing time: 86 msecs > [junit4] 2> 819712 INFO (IPC Server handler 3 on 59517) [ ] > BlockStateChange BLOCK* processReport: from storage > DS-394f120c-9225-45db-9d4d-a79e0357d0a0 node > DatanodeRegistration(127.0.0.1:38537, > datanodeUuid=6109fe6a-418b-4810-8965-110624cd978c, infoPort=48162, > infoSecurePort=0, ipcPort=33212, > storageInfo=lv=-56;cid=testClusterID;nsid=2101821981;c=0), blocks: 0, > hasStaleStorage: false, processing time: 0 msecs > [junit4] 2> 819976 INFO > (TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [ ] > o.a.s.c.ZkTestServer STARTING ZK TEST SERVER > [junit4] 2> 819977 INFO (Thread-1303) [ ] o.a.s.c.ZkTestServer client > port:0.0.0.0/0.0.0.0:0 > [junit4] 2> 819977 INFO (Thread-1303) [ ] o.a.s.c.ZkTestServer > Starting server > [junit4] 2> 820180 INFO > (TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [ ] > o.a.s.c.ZkTestServer start zk server on port:43044 > [junit4] 2> 820180 INFO > (TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [ ] > o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider > [junit4] 2> 820189 INFO > (TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [ ] > o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper > [junit4] 2> 820201 INFO (zkCallback-698-thread-1) [ ] > o.a.s.c.c.ConnectionManager Watcher > org.apache.solr.common.cloud.ConnectionManager@6ea7009c > name:ZooKeeperConnection Watcher:127.0.0.1:43044 got event WatchedEvent > state:SyncConnected type:None path:null path:null type:None > [junit4] 2> 820201 INFO > (TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [ ] > o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper > [junit4] 2> 820201 INFO > (TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [ ] > o.a.s.c.c.SolrZkClient Using default ZkACLProvider > [junit4] 2> 820201 INFO > (TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [ ] > o.a.s.c.c.SolrZkClient makePath: /solr > [junit4] 2> 820210 INFO > (TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [ ] > o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider > [junit4] 2> 820215 INFO > (TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [ ] > o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper > [junit4] 2> 820239 INFO (zkCallback-699-thread-1) [ ] > o.a.s.c.c.ConnectionManager Watcher > org.apache.solr.common.cloud.ConnectionManager@1095833c > name:ZooKeeperConnection Watcher:127.0.0.1:43044/solr got event WatchedEvent > state:SyncConnected type:None path:null path:null type:None > [junit4] 2> 820241 INFO > (TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [ ] > o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper > [junit4] 2> 820241 INFO > (TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [ ] > o.a.s.c.c.SolrZkClient Using default ZkACLProvider > [junit4] 2> 820241 INFO > (TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [ ] > o.a.s.c.c.SolrZkClient makePath: /collections/collection1 > [junit4] 2> 820242 INFO > (TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [ ] > o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards > [junit4] 2> 820244 INFO > (TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [ ] > o.a.s.c.c.SolrZkClient makePath: /collections/control_collection > [junit4] 2> 820244 INFO > (TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [ ] > o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards > [junit4] 2> 820250 INFO > (TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [ ] > o.a.s.c.AbstractZkTestCase put > /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml > to /configs/conf1/solrconfig.xml > [junit4] 2> 820250 INFO > (TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [ ] > o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml > [junit4] 2> 820252 INFO > (TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [ ] > o.a.s.c.AbstractZkTestCase put > /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/schema.xml > to /configs/conf1/schema.xml > [junit4] 2> 820252 INFO > (TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [ ] > o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml > [junit4] 2> 820253 INFO > (TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [ ] > o.a.s.c.AbstractZkTestCase put > /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml > to /configs/conf1/solrconfig.snippet.randomindexconfig.xml > [junit4] 2> 820253 INFO > (TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [ ] > o.a.s.c.c.SolrZkClient makePath: > /configs/conf1/solrconfig.snippet.randomindexconfig.xml > [junit4] 2> 820255 INFO > (TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [ ] > o.a.s.c.AbstractZkTestCase put > /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt > to /configs/conf1/stopwords.txt > [junit4] 2> 820255 INFO > (TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [ ] > o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt > [junit4] 2> 820262 INFO > (TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [ ] > o.a.s.c.AbstractZkTestCase put > /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt > to /configs/conf1/protwords.txt > [junit4] 2> 820262 INFO > (TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [ ] > o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt > [junit4] 2> 820263 INFO > (TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [ ] > o.a.s.c.AbstractZkTestCase put > /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml > to /configs/conf1/currency.xml > [junit4] 2> 820263 INFO > (TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [ ] > o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml > [junit4] 2> 820264 INFO > (TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [ ] > o.a.s.c.AbstractZkTestCase put > /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml > to /configs/conf1/enumsConfig.xml > [junit4] 2> 820264 INFO > (TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [ ] > o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml > [junit4] 2> 820270 INFO > (TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [ ] > o.a.s.c.AbstractZkTestCase put /x1/jenki > > [...truncated too long message...] > > ctory Closing directory, CoreContainer#isShutdown=true > [junit4] 2> 4696298 INFO (coreCloseExecutor-5736-thread-1) [ > x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory: > /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedFacetPivotLongTailTest_66DE89DEA79A3E32-001/tempDir-001/shard2/cores/collection1/data/index > [junit4] 2> 4696305 INFO > (TEST-DistributedFacetPivotLongTailTest.test-seed#[66DE89DEA79A3E32]) [ ] > o.e.j.s.h.ContextHandler Stopped > o.e.j.s.ServletContextHandler@17e66fe8{/_/d,null,UNAVAILABLE} > [junit4] 2> NOTE: download the large Jenkins line-docs file by running > 'ant get-jenkins-line-docs' in the lucene directory. > [junit4] 2> NOTE: reproduce with: ant test > -Dtestcase=DistributedFacetPivotLongTailTest -Dtests.method=test > -Dtests.seed=66DE89DEA79A3E32 -Dtests.multiplier=2 -Dtests.nightly=true > -Dtests.slow=true > -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt > -Dtests.locale=cs -Dtests.timezone=Etc/GMT-6 -Dtests.asserts=true > -Dtests.file.encoding=US-ASCII > [junit4] ERROR 131s J0 | DistributedFacetPivotLongTailTest.test <<< > [junit4] > Throwable #1: > org.apache.solr.client.solrj.SolrServerException: IOException occured when > talking to server at: https://127.0.0.1:46729/_/d/collection1 > [junit4] > at > __randomizedtesting.SeedInfo.seed([66DE89DEA79A3E32:EE8AB604096653CA]:0) > [junit4] > at > org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:622) > [junit4] > at > org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261) > [junit4] > at > org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250) > [junit4] > at > org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149) > [junit4] > at > org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173) > [junit4] > at > org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138) > [junit4] > at > org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:152) > [junit4] > at > org.apache.solr.handler.component.DistributedFacetPivotLongTailTest.test(DistributedFacetPivotLongTailTest.java:71) > [junit4] > at > org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985) > [junit4] > at > org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960) > [junit4] > at java.lang.Thread.run(Thread.java:745) > [junit4] > Caused by: org.apache.http.conn.ConnectTimeoutException: > Connect to 127.0.0.1:46729 [/127.0.0.1] failed: Read timed out > [junit4] > at > org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:143) > [junit4] > at > org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) > [junit4] > at > org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) > [junit4] > at > org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) > [junit4] > at > org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) > [junit4] > at > org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) > [junit4] > at > org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) > [junit4] > at > org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) > [junit4] > at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) > [junit4] > at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) > [junit4] > at > org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:513) > [junit4] > ... 48 more > [junit4] > Caused by: java.net.SocketTimeoutException: Read timed out > [junit4] > at java.net.SocketInputStream.socketRead0(Native Method) > [junit4] > at > java.net.SocketInputStream.socketRead(SocketInputStream.java:116) > [junit4] > at > java.net.SocketInputStream.read(SocketInputStream.java:170) > [junit4] > at > java.net.SocketInputStream.read(SocketInputStream.java:141) > [junit4] > at > sun.security.ssl.InputRecord.readFully(InputRecord.java:465) > [junit4] > at > sun.security.ssl.InputRecord.read(InputRecord.java:503) > [junit4] > at > sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) > [junit4] > at > sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375) > [junit4] > at > sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403) > [junit4] > at > sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387) > [junit4] > at > org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394) > [junit4] > at > org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353) > [junit4] > at > org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134) > [junit4] > ... 58 more > [junit4] 2> 4696316 INFO > (SUITE-DistributedFacetPivotLongTailTest-seed#[66DE89DEA79A3E32]-worker) [ > ] o.a.s.SolrTestCaseJ4 ###deleteCore > [junit4] 2> NOTE: leaving temporary files on disk at: > /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedFacetPivotLongTailTest_66DE89DEA79A3E32-001 > [junit4] 2> NOTE: test params are: codec=Lucene62, sim=ClassicSimilarity, > locale=cs, timezone=Etc/GMT-6 > [junit4] 2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation > 1.8.0_102 (64-bit)/cpus=4,threads=1,free=251613816,total=477626368 > [junit4] 2> NOTE: All tests run in this JVM: > [OverseerModifyCollectionTest, RequiredFieldsTest, > DistributedDebugComponentTest, TestMissingGroups, TestIBSimilarityFactory, > CdcrBootstrapTest, TestManagedSynonymFilterFactory, > LeaderInitiatedRecoveryOnShardRestartTest, TestSubQueryTransformerCrossCore, > TestOnReconnectListenerSupport, ZkStateReaderTest, > TestSolrQueryParserResource, ResourceLoaderTest, TestConfigSetImmutable, > TestSolr4Spatial, TestFreeTextSuggestions, TestReplicationHandlerBackup, > ReturnFieldsTest, TestClassicSimilarityFactory, HdfsThreadLeakTest, > SharedFSAutoReplicaFailoverUtilsTest, TestCSVLoader, > LeaderElectionIntegrationTest, ZkNodePropsTest, TestRemoteStreaming, > TestSchemaManager, EnumFieldTest, BasicZkTest, DirectoryFactoryTest, > RemoteQueryErrorTest, SOLR749Test, TestLMDirichletSimilarityFactory, > TestStressUserVersions, TestFieldCacheSortRandom, TestDistribDocBasedVersion, > ChangedSchemaMergeTest, RuleEngineTest, TestPhraseSuggestions, > TestCloudPseudoReturnFields, TestReplicationHandler, > DisMaxRequestHandlerTest, TestLegacyFieldCache, > TestComplexPhraseQParserPlugin, BasicDistributedZkTest, TestLRUCache, > TestDistributedSearch, RestartWhileUpdatingTest, TestMergePolicyConfig, > TestRebalanceLeaders, TestInitParams, CacheHeaderTest, BitVectorTest, > TestArbitraryIndexDir, TestCloudDeleteByQuery, > PathHierarchyTokenizerFactoryTest, TestSchemaNameResource, TestOrdValues, > DistributedFacetPivotLargeTest, BlockJoinFacetDistribTest, TestFaceting, > SynonymTokenizerTest, SpatialHeatmapFacetsTest, TestMiniSolrCloudClusterBase, > OutOfBoxZkACLAndCredentialsProvidersTest, QueryEqualityTest, IndexSchemaTest, > TestBadConfig, TestAddFieldRealTimeGet, BlockDirectoryTest, > DefaultValueUpdateProcessorTest, TestCollationFieldDocValues, > TestFoldingMultitermQuery, TestBlobHandler, SolrCloudExampleTest, > DistributedSpellCheckComponentTest, ConnectionReuseTest, > PingRequestHandlerTest, DocumentAnalysisRequestHandlerTest, > AddSchemaFieldsUpdateProcessorFactoryTest, > HdfsCollectionsAPIDistributedZkTest, TestRTGBase, > CloneFieldUpdateProcessorFactoryTest, TestSizeLimitedDistributedMap, > TestLuceneMatchVersion, TestRawResponseWriter, > SignatureUpdateProcessorFactoryTest, TestXmlQParserPlugin, > TestComponentsName, TestSQLHandler, HdfsLockFactoryTest, > TestIntervalFaceting, TestCloudManagedSchema, > TestDynamicFieldCollectionResource, TestSortingResponseWriter, > TestHighlightDedupGrouping, UUIDFieldTest, ConnectionManagerTest, > AnalysisAfterCoreReloadTest, TestSchemaVersionResource, JSONWriterTest, > TestUninvertingReader, TermVectorComponentDistributedTest, > TestGroupingSearch, SimpleCollectionCreateDeleteTest, > DistributedFacetPivotLongTailTest] > [junit4] Completed [632/633 (5!)] on J0 in 130.67s, 1 test, 1 error <<< > FAILURES! > > [...truncated 1 lines...] > [junit4] JVM J0: stdout was not empty, see: > /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/temp/junit4-J0-20160907_083928_132.sysout > [junit4] >>> JVM J0 emitted unexpected output (verbatim) ---- > [junit4] java.lang.OutOfMemoryError: GC overhead limit exceeded > [junit4] Dumping heap to > /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/heapdumps/java_pid30315.hprof > ... > [junit4] Heap dump file created [707246725 bytes in 19.813 secs] > [junit4] <<< JVM J0: EOF ---- > > [...truncated 11039 lines...] > BUILD FAILED > /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:771: > The following error occurred while executing this line: > /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:715: > Some of the tests produced a heap dump, but did not fail. Maybe a suppressed > OutOfMemoryError? Dumps created: > * java_pid30315.hprof > > Total time: 367 minutes 35 seconds > Build step 'Invoke Ant' marked build as failure > Archiving artifacts > Recording test results > Email was triggered for: Failure - Any > Sending email for trigger: Failure - Any > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
