Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1111/

10 tests failed.
FAILED:  org.apache.lucene.index.TestIndexWriterReader.testDuringAddDelete

Error Message:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/J2/temp/lucene.index.TestIndexWriterReader_61A624ED3323A1DA-001/index-SimpleFSDirectory-001/_24x_Lucene50_1.tim:
 Too many open files

Stack Trace:
java.nio.file.FileSystemException: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/J2/temp/lucene.index.TestIndexWriterReader_61A624ED3323A1DA-001/index-SimpleFSDirectory-001/_24x_Lucene50_1.tim:
 Too many open files
        at 
__randomizedtesting.SeedInfo.seed([61A624ED3323A1DA:FF37536A19E223DA]:0)
        at 
org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48)
        at 
org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81)
        at 
org.apache.lucene.mockfile.HandleTrackingFS.newByteChannel(HandleTrackingFS.java:271)
        at 
org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212)
        at 
org.apache.lucene.mockfile.HandleTrackingFS.newByteChannel(HandleTrackingFS.java:240)
        at 
org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212)
        at java.nio.file.Files.newByteChannel(Files.java:361)
        at java.nio.file.Files.newByteChannel(Files.java:407)
        at 
org.apache.lucene.store.SimpleFSDirectory.openInput(SimpleFSDirectory.java:77)
        at 
org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:99)
        at 
org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.<init>(BlockTreeTermsReader.java:153)
        at 
org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat.fieldsProducer(Lucene50PostingsFormat.java:445)
        at 
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:261)
        at 
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:341)
        at 
org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:106)
        at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:74)
        at 
org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145)
        at 
org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:197)
        at 
org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:103)
        at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:460)
        at 
org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:291)
        at 
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:266)
        at 
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:256)
        at 
org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140)
        at 
org.apache.lucene.index.TestIndexWriterReader.testDuringAddDelete(TestIndexWriterReader.java:870)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.lucene.index.TestIndexWriterReader

Error Message:
5 threads leaked from SUITE scope at 
org.apache.lucene.index.TestIndexWriterReader:     1) Thread[id=1799, 
name=Thread-1549, state=BLOCKED, group=TGRP-TestIndexWriterReader]         at 
org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37)
         at 
org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052)         
at 
org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910)
         at 
org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
         at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948)        
 at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939)    
     at 
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565)       
  at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307)     
    at 
org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848)
    2) Thread[id=1801, name=Thread-1551, state=RUNNABLE, 
group=TGRP-TestIndexWriterReader]         at 
org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.get(CompressingTermVectorsReader.java:417)
         at 
org.apache.lucene.codecs.asserting.AssertingTermVectorsFormat$AssertingTermVectorsReader.get(AssertingTermVectorsFormat.java:71)
         at 
org.apache.lucene.codecs.TermVectorsWriter.merge(TermVectorsWriter.java:223)    
     at 
org.apache.lucene.index.SegmentMerger.mergeVectors(SegmentMerger.java:210)      
   at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:144)       
  at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4287)     
    at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3864)         
at 
org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
         at 
org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052)         
at 
org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910)
         at 
org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
         at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948)        
 at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939)    
     at 
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565)       
  at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307)     
    at 
org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848)
    3) Thread[id=1802, name=Thread-1552, state=BLOCKED, 
group=TGRP-TestIndexWriterReader]         at 
org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37)
         at 
org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052)         
at 
org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910)
         at 
org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
         at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948)        
 at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939)    
     at 
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565)       
  at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307)     
    at 
org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848)
    4) Thread[id=1800, name=Thread-1550, state=BLOCKED, 
group=TGRP-TestIndexWriterReader]         at 
org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37)
         at 
org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052)         
at 
org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910)
         at 
org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
         at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948)        
 at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939)    
     at 
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565)       
  at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307)     
    at 
org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848)
    5) Thread[id=1798, name=Thread-1548, state=BLOCKED, 
group=TGRP-TestIndexWriterReader]         at 
org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37)
         at 
org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052)         
at 
org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910)
         at 
org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
         at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948)        
 at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939)    
     at 
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565)       
  at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307)     
    at 
org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from SUITE 
scope at org.apache.lucene.index.TestIndexWriterReader: 
   1) Thread[id=1799, name=Thread-1549, state=BLOCKED, 
group=TGRP-TestIndexWriterReader]
        at 
org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37)
        at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052)
        at 
org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910)
        at 
org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
        at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948)
        at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939)
        at 
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565)
        at 
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307)
        at 
org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848)
   2) Thread[id=1801, name=Thread-1551, state=RUNNABLE, 
group=TGRP-TestIndexWriterReader]
        at 
org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.get(CompressingTermVectorsReader.java:417)
        at 
org.apache.lucene.codecs.asserting.AssertingTermVectorsFormat$AssertingTermVectorsReader.get(AssertingTermVectorsFormat.java:71)
        at 
org.apache.lucene.codecs.TermVectorsWriter.merge(TermVectorsWriter.java:223)
        at 
org.apache.lucene.index.SegmentMerger.mergeVectors(SegmentMerger.java:210)
        at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:144)
        at 
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4287)
        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3864)
        at 
org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
        at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052)
        at 
org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910)
        at 
org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
        at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948)
        at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939)
        at 
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565)
        at 
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307)
        at 
org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848)
   3) Thread[id=1802, name=Thread-1552, state=BLOCKED, 
group=TGRP-TestIndexWriterReader]
        at 
org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37)
        at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052)
        at 
org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910)
        at 
org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
        at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948)
        at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939)
        at 
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565)
        at 
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307)
        at 
org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848)
   4) Thread[id=1800, name=Thread-1550, state=BLOCKED, 
group=TGRP-TestIndexWriterReader]
        at 
org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37)
        at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052)
        at 
org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910)
        at 
org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
        at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948)
        at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939)
        at 
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565)
        at 
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307)
        at 
org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848)
   5) Thread[id=1798, name=Thread-1548, state=BLOCKED, 
group=TGRP-TestIndexWriterReader]
        at 
org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37)
        at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052)
        at 
org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910)
        at 
org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
        at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948)
        at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939)
        at 
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565)
        at 
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307)
        at 
org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848)
        at __randomizedtesting.SeedInfo.seed([61A624ED3323A1DA]:0)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.CollectionsAPIDistributedZkTest

Error Message:
1 thread leaked from SUITE scope at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest:     1) Thread[id=125227, 
name=searcherExecutor-10822-thread-1, state=WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE 
scope at org.apache.solr.cloud.CollectionsAPIDistributedZkTest: 
   1) Thread[id=125227, name=searcherExecutor-10822-thread-1, state=WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        at __randomizedtesting.SeedInfo.seed([66DE89DEA79A3E32]:0)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.CollectionsAPIDistributedZkTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) 
Thread[id=125227, name=searcherExecutor-10822-thread-1, state=WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie 
threads that couldn't be terminated:
   1) Thread[id=125227, name=searcherExecutor-10822-thread-1, state=WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        at __randomizedtesting.SeedInfo.seed([66DE89DEA79A3E32]:0)


FAILED:  org.apache.solr.cloud.hdfs.StressHdfsTest.test

Error Message:
Could not find collection:delete_data_dir

Stack Trace:
java.lang.AssertionError: Could not find collection:delete_data_dir
        at 
__randomizedtesting.SeedInfo.seed([66DE89DEA79A3E32:EE8AB604096653CA]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNotNull(Assert.java:526)
        at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:154)
        at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:139)
        at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:134)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:856)
        at 
org.apache.solr.cloud.hdfs.StressHdfsTest.createAndDeleteCollection(StressHdfsTest.java:159)
        at 
org.apache.solr.cloud.hdfs.StressHdfsTest.test(StressHdfsTest.java:103)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: 
https://127.0.0.1:39500/vum

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting 
response from server at: https://127.0.0.1:39500/vum
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:619)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:400)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:516)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:180)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:170)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
        at sun.security.ssl.InputRecord.read(InputRecord.java:503)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
        at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
        at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
        at 
org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
        at 
org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
        at 
org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
        at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
        at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
        at 
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
        at 
org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
        at 
org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
        at 
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
        at 
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
        at 
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
        at 
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
        at 
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at 
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:513)
        ... 47 more


FAILED:  
org.apache.solr.handler.component.DistributedFacetPivotLongTailTest.test

Error Message:
IOException occured when talking to server at: 
https://127.0.0.1:46729/_/d/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: IOException occured when 
talking to server at: https://127.0.0.1:46729/_/d/collection1
        at 
__randomizedtesting.SeedInfo.seed([66DE89DEA79A3E32:EE8AB604096653CA]:0)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:622)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
        at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173)
        at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138)
        at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:152)
        at 
org.apache.solr.handler.component.DistributedFacetPivotLongTailTest.test(DistributedFacetPivotLongTailTest.java:71)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to 
127.0.0.1:46729 [/127.0.0.1] failed: Read timed out
        at 
org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:143)
        at 
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
        at 
org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
        at 
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
        at 
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
        at 
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at 
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:513)
        ... 48 more
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:170)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
        at sun.security.ssl.InputRecord.read(InputRecord.java:503)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
        at 
sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
        at 
sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
        at 
sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
        at 
org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394)
        at 
org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353)
        at 
org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
        ... 58 more


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: https://127.0.0.1:35113

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting 
response from server at: https://127.0.0.1:35113
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:619)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:400)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:516)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:180)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:170)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
        at sun.security.ssl.InputRecord.read(InputRecord.java:503)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
        at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
        at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
        at 
org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
        at 
org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
        at 
org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
        at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
        at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
        at 
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
        at 
org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
        at 
org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
        at 
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
        at 
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
        at 
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
        at 
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
        at 
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at 
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:513)
        ... 47 more


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest

Error Message:
ObjectTracker found 10 object(s) that were not released!!! [InternalHttpClient, 
InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, 
InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, 
InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 10 object(s) that were not 
released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, 
InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, 
InternalHttpClient, InternalHttpClient, InternalHttpClient]
        at __randomizedtesting.SeedInfo.seed([66DE89DEA79A3E32]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNull(Assert.java:551)
        at 
org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
        at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest

Error Message:
10 threads leaked from SUITE scope at 
org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest:     1) 
Thread[id=28289, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=28294, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=28264, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=28259, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=28269, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=28304, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=28284, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=28299, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=28274, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=28279, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from 
SUITE scope at org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest: 
   1) Thread[id=28289, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=28294, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=28264, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=28259, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=28269, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=28304, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=28284, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=28299, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   9) Thread[id=28274, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  10) Thread[id=28279, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
        at __randomizedtesting.SeedInfo.seed([66DE89DEA79A3E32]:0)




Build Log:
[...truncated 913 lines...]
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterReader
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 
'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=TestIndexWriterReader -Dtests.method=testDuringAddDelete 
-Dtests.seed=61A624ED3323A1DA -Dtests.multiplier=2 -Dtests.nightly=true 
-Dtests.slow=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt
 -Dtests.locale=mk -Dtests.timezone=Europe/Skopje -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8
   [junit4] ERROR    142s J2 | TestIndexWriterReader.testDuringAddDelete <<<
   [junit4]    > Throwable #1: java.nio.file.FileSystemException: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/J2/temp/lucene.index.TestIndexWriterReader_61A624ED3323A1DA-001/index-SimpleFSDirectory-001/_24x_Lucene50_1.tim:
 Too many open files
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([61A624ED3323A1DA:FF37536A19E223DA]:0)
   [junit4]    >        at 
org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48)
   [junit4]    >        at 
org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81)
   [junit4]    >        at 
org.apache.lucene.mockfile.HandleTrackingFS.newByteChannel(HandleTrackingFS.java:271)
   [junit4]    >        at 
org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212)
   [junit4]    >        at 
org.apache.lucene.mockfile.HandleTrackingFS.newByteChannel(HandleTrackingFS.java:240)
   [junit4]    >        at 
org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212)
   [junit4]    >        at java.nio.file.Files.newByteChannel(Files.java:361)
   [junit4]    >        at java.nio.file.Files.newByteChannel(Files.java:407)
   [junit4]    >        at 
org.apache.lucene.store.SimpleFSDirectory.openInput(SimpleFSDirectory.java:77)
   [junit4]    >        at 
org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:99)
   [junit4]    >        at 
org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.<init>(BlockTreeTermsReader.java:153)
   [junit4]    >        at 
org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat.fieldsProducer(Lucene50PostingsFormat.java:445)
   [junit4]    >        at 
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:261)
   [junit4]    >        at 
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:341)
   [junit4]    >        at 
org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:106)
   [junit4]    >        at 
org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:74)
   [junit4]    >        at 
org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145)
   [junit4]    >        at 
org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:197)
   [junit4]    >        at 
org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:103)
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:460)
   [junit4]    >        at 
org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:291)
   [junit4]    >        at 
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:266)
   [junit4]    >        at 
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:256)
   [junit4]    >        at 
org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140)
   [junit4]    >        at 
org.apache.lucene.index.TestIndexWriterReader.testDuringAddDelete(TestIndexWriterReader.java:870)
   [junit4]    >        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/J2/temp/lucene.index.TestIndexWriterReader_61A624ED3323A1DA-001
   [junit4]   2> Sep 07, 2016 4:42:35 AM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 5 leaked 
thread(s).
   [junit4]   2> Sep 07, 2016 4:42:55 AM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 5 threads leaked from SUITE scope at 
org.apache.lucene.index.TestIndexWriterReader: 
   [junit4]   2>    1) Thread[id=1799, name=Thread-1549, state=BLOCKED, 
group=TGRP-TestIndexWriterReader]
   [junit4]   2>         at 
org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910)
   [junit4]   2>         at 
org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307)
   [junit4]   2>         at 
org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848)
   [junit4]   2>    2) Thread[id=1801, name=Thread-1551, state=RUNNABLE, 
group=TGRP-TestIndexWriterReader]
   [junit4]   2>         at 
org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.get(CompressingTermVectorsReader.java:417)
   [junit4]   2>         at 
org.apache.lucene.codecs.asserting.AssertingTermVectorsFormat$AssertingTermVectorsReader.get(AssertingTermVectorsFormat.java:71)
   [junit4]   2>         at 
org.apache.lucene.codecs.TermVectorsWriter.merge(TermVectorsWriter.java:223)
   [junit4]   2>         at 
org.apache.lucene.index.SegmentMerger.mergeVectors(SegmentMerger.java:210)
   [junit4]   2>         at 
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:144)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4287)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3864)
   [junit4]   2>         at 
org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910)
   [junit4]   2>         at 
org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307)
   [junit4]   2>         at 
org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848)
   [junit4]   2>    3) Thread[id=1802, name=Thread-1552, state=BLOCKED, 
group=TGRP-TestIndexWriterReader]
   [junit4]   2>         at 
org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910)
   [junit4]   2>         at 
org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307)
   [junit4]   2>         at 
org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848)
   [junit4]   2>    4) Thread[id=1800, name=Thread-1550, state=BLOCKED, 
group=TGRP-TestIndexWriterReader]
   [junit4]   2>         at 
org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910)
   [junit4]   2>         at 
org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307)
   [junit4]   2>         at 
org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848)
   [junit4]   2>    5) Thread[id=1798, name=Thread-1548, state=BLOCKED, 
group=TGRP-TestIndexWriterReader]
   [junit4]   2>         at 
org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910)
   [junit4]   2>         at 
org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307)
   [junit4]   2>         at 
org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848)
   [junit4]   2> Sep 07, 2016 4:42:55 AM 
com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=1799, name=Thread-1549, state=BLOCKED, 
group=TGRP-TestIndexWriterReader]
   [junit4]   2>    2) Thread[id=1801, name=Thread-1551, state=RUNNABLE, 
group=TGRP-TestIndexWriterReader]
   [junit4]   2>    3) Thread[id=1802, name=Thread-1552, state=BLOCKED, 
group=TGRP-TestIndexWriterReader]
   [junit4]   2>    4) Thread[id=1800, name=Thread-1550, state=BLOCKED, 
group=TGRP-TestIndexWriterReader]
   [junit4]   2>    5) Thread[id=1798, name=Thread-1548, state=BLOCKED, 
group=TGRP-TestIndexWriterReader]
   [junit4]   2> Sep 07, 2016 4:42:55 AM 
com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: All leaked threads terminated.
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62): 
{field1=Lucene50(blocksize=128), field=Lucene50(blocksize=128), 
indexname=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))),
 foo=Lucene50(blocksize=128), 
id=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))),
 
field3=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))),
 field2=FSTOrd50, field5=Lucene50(blocksize=128), 
field4=Lucene50(blocksize=128)}, docValues:{}, maxPointsInLeafNode=1314, 
maxMBSortInHeap=5.621346163809926, sim=RandomSimilarity(queryNorm=true): 
{field1=DFR I(F)B1, field=DFR I(n)L2, indexname=DFR GLZ(0.3), foo=DFR GLZ(0.3), 
field3=DFR I(ne)Z(0.3), field2=IB SPL-D1, field5=DFR I(F)1, field4=DFR 
I(F)LZ(0.3)}, locale=mk, timezone=Europe/Skopje
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 
1.8.0_102 (64-bit)/cpus=4,threads=1,free=139897912,total=268959744
   [junit4]   2> NOTE: All tests run in this JVM: [Test2BPagedBytes, 
TestStressIndexing, TestGeoUtils, TestNoMergePolicy, TestRollback, 
TestBlockPostingsFormat, TestTermQuery, TestTopDocsMerge, TestFilterIterator, 
TestStressDeletes, TestIntsRef, TestDocBoost, TestNamedSPILoader, 
TestTransactions, TestRollingBuffer, TestFilterWeight, 
TestIndexWriterThreadsToSegments, TestLogMergePolicy, TestHighCompressionMode, 
TestDocValuesIndexing, TestRegexpRandom2, TestSingleInstanceLockFactory, 
TestLucene62SegmentInfoFormat, TestSegmentMerger, TestIndexWriterCommit, 
TestNRTCachingDirectory, TestAllFilesHaveCodecHeader, TestFuzzyQuery, 
Test2BPostings, TestDocumentsWriterDeleteQueue, TestByteBlockPool, 
TestScorerPerf, TestPayloadsOnVectors, TestParallelLeafReader, 
TestSegmentTermDocs, TestParallelTermEnum, TestDocument, TestVersion, 
TestAtomicUpdate, TestConjunctionDISI, TestBagOfPostings, 
TestMultiTermConstantScore, TestSnapshotDeletionPolicy, TestMmapDirectory, 
TestSpansEnum, TestSpanNotQuery, TestTimSorter, TestWildcardRandom, 
TestInPlaceMergeSorter, TestFixedBitDocIdSet, TestSimilarity, 
TestSearchWithThreads, TestCharacterUtils, TestUpgradeIndexMergePolicy, 
TestBlendedTermQuery, TestBooleanQuery, TestArrayUtil, 
TestSimpleSearchEquivalence, TestSpanMultiTermQueryWrapper, TestDeletionPolicy, 
TestMultiPhraseEnum, TestLazyProxSkipping, TestForUtil, TestSpanExplanations, 
TestReaderWrapperDVTypeCheck, TestFastCompressionMode, TestRegexpQuery, 
TestTopFieldCollector, TestMSBRadixSorter, TestIndexWriterMerging, 
TestFileSwitchDirectory, TestIndexWriter, TestIndexWriterReader]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 
'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=TestIndexWriterReader -Dtests.seed=61A624ED3323A1DA 
-Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt
 -Dtests.locale=mk -Dtests.timezone=Europe/Skopje -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J2 | TestIndexWriterReader (suite) <<<
   [junit4]    > Throwable #1: 
com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from SUITE 
scope at org.apache.lucene.index.TestIndexWriterReader: 
   [junit4]    >    1) Thread[id=1799, name=Thread-1549, state=BLOCKED, 
group=TGRP-TestIndexWriterReader]
   [junit4]    >         at 
org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37)
   [junit4]    >         at 
org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052)
   [junit4]    >         at 
org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910)
   [junit4]    >         at 
org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
   [junit4]    >         at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948)
   [junit4]    >         at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939)
   [junit4]    >         at 
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565)
   [junit4]    >         at 
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307)
   [junit4]    >         at 
org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848)
   [junit4]    >    2) Thread[id=1801, name=Thread-1551, state=RUNNABLE, 
group=TGRP-TestIndexWriterReader]
   [junit4]    >         at 
org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.get(CompressingTermVectorsReader.java:417)
   [junit4]    >         at 
org.apache.lucene.codecs.asserting.AssertingTermVectorsFormat$AssertingTermVectorsReader.get(AssertingTermVectorsFormat.java:71)
   [junit4]    >         at 
org.apache.lucene.codecs.TermVectorsWriter.merge(TermVectorsWriter.java:223)
   [junit4]    >         at 
org.apache.lucene.index.SegmentMerger.mergeVectors(SegmentMerger.java:210)
   [junit4]    >         at 
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:144)
   [junit4]    >         at 
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4287)
   [junit4]    >         at 
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3864)
   [junit4]    >         at 
org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
   [junit4]    >         at 
org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052)
   [junit4]    >         at 
org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910)
   [junit4]    >         at 
org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
   [junit4]    >         at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948)
   [junit4]    >         at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939)
   [junit4]    >         at 
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565)
   [junit4]    >         at 
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307)
   [junit4]    >         at 
org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848)
   [junit4]    >    3) Thread[id=1802, name=Thread-1552, state=BLOCKED, 
group=TGRP-TestIndexWriterReader]
   [junit4]    >         at 
org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37)
   [junit4]    >         at 
org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052)
   [junit4]    >         at 
org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910)
   [junit4]    >         at 
org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
   [junit4]    >         at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948)
   [junit4]    >         at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939)
   [junit4]    >         at 
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565)
   [junit4]    >         at 
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307)
   [junit4]    >         at 
org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848)
   [junit4]    >    4) Thread[id=1800, name=Thread-1550, state=BLOCKED, 
group=TGRP-TestIndexWriterReader]
   [junit4]    >         at 
org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37)
   [junit4]    >         at 
org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052)
   [junit4]    >         at 
org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910)
   [junit4]    >         at 
org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
   [junit4]    >         at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948)
   [junit4]    >         at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939)
   [junit4]    >         at 
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565)
   [junit4]    >         at 
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307)
   [junit4]    >         at 
org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848)
   [junit4]    >    5) Thread[id=1798, name=Thread-1548, state=BLOCKED, 
group=TGRP-TestIndexWriterReader]
   [junit4]    >         at 
org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37)
   [junit4]    >         at 
org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052)
   [junit4]    >         at 
org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4910)
   [junit4]    >         at 
org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
   [junit4]    >         at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4948)
   [junit4]    >         at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4939)
   [junit4]    >         at 
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565)
   [junit4]    >         at 
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307)
   [junit4]    >         at 
org.apache.lucene.index.TestIndexWriterReader$2.run(TestIndexWriterReader.java:848)
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([61A624ED3323A1DA]:0)
   [junit4] Completed [200/432 (1!)] on J2 in 252.56s, 23 tests, 2 errors <<< 
FAILURES!

[...truncated 10658 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.StressHdfsTest
   [junit4]   2> Creating dataDir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_66DE89DEA79A3E32-001/init-core-data-001
   [junit4]   2> 814899 INFO  
(SUITE-StressHdfsTest-seed#[66DE89DEA79A3E32]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: 
@org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 814899 INFO  
(SUITE-StressHdfsTest-seed#[66DE89DEA79A3E32]-worker) [    ] 
o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /_a/m
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 815026 WARN  
(SUITE-StressHdfsTest-seed#[66DE89DEA79A3E32]-worker) [    ] 
o.a.h.m.i.MetricsConfig Cannot locate configuration: tried 
hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 815069 WARN  
(SUITE-StressHdfsTest-seed#[66DE89DEA79A3E32]-worker) [    ] 
o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 815081 INFO  
(SUITE-StressHdfsTest-seed#[66DE89DEA79A3E32]-worker) [    ] o.m.log 
jetty-6.1.26
   [junit4]   2> 815125 INFO  
(SUITE-StressHdfsTest-seed#[66DE89DEA79A3E32]-worker) [    ] o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs
 to ./temp/Jetty_localhost_33702_hdfs____13ay83/webapp
   [junit4]   2> 816226 INFO  
(SUITE-StressHdfsTest-seed#[66DE89DEA79A3E32]-worker) [    ] o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33702
   [junit4]   2> 817177 WARN  
(SUITE-StressHdfsTest-seed#[66DE89DEA79A3E32]-worker) [    ] 
o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 817179 INFO  
(SUITE-StressHdfsTest-seed#[66DE89DEA79A3E32]-worker) [    ] o.m.log 
jetty-6.1.26
   [junit4]   2> 817200 INFO  
(SUITE-StressHdfsTest-seed#[66DE89DEA79A3E32]-worker) [    ] o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode
 to ./temp/Jetty_localhost_46024_datanode____.2mcfop/webapp
   [junit4]   2> 818238 INFO  
(SUITE-StressHdfsTest-seed#[66DE89DEA79A3E32]-worker) [    ] o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46024
   [junit4]   2> 818469 WARN  
(SUITE-StressHdfsTest-seed#[66DE89DEA79A3E32]-worker) [    ] 
o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 818470 INFO  
(SUITE-StressHdfsTest-seed#[66DE89DEA79A3E32]-worker) [    ] o.m.log 
jetty-6.1.26
   [junit4]   2> 818502 INFO  
(SUITE-StressHdfsTest-seed#[66DE89DEA79A3E32]-worker) [    ] o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode
 to ./temp/Jetty_localhost_55265_datanode____.v9bv0o/webapp
   [junit4]   2> 818789 INFO  (IPC Server handler 6 on 59517) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-124aedd7-53e1-4abe-9593-1749020619f8 node 
DatanodeRegistration(127.0.0.1:39806, 
datanodeUuid=44048ab0-2317-4266-b18a-947e4bc5cd29, infoPort=51796, 
infoSecurePort=0, ipcPort=39160, 
storageInfo=lv=-56;cid=testClusterID;nsid=2101821981;c=0), blocks: 0, 
hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 818789 INFO  (IPC Server handler 6 on 59517) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-0427d4bd-1eeb-4af5-a408-0b312310f3bb node 
DatanodeRegistration(127.0.0.1:39806, 
datanodeUuid=44048ab0-2317-4266-b18a-947e4bc5cd29, infoPort=51796, 
infoSecurePort=0, ipcPort=39160, 
storageInfo=lv=-56;cid=testClusterID;nsid=2101821981;c=0), blocks: 0, 
hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 819366 INFO  
(SUITE-StressHdfsTest-seed#[66DE89DEA79A3E32]-worker) [    ] o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:55265
   [junit4]   2> 819712 INFO  (IPC Server handler 3 on 59517) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-0a413b87-a6df-47f1-b731-f3928bdb7bd0 node 
DatanodeRegistration(127.0.0.1:38537, 
datanodeUuid=6109fe6a-418b-4810-8965-110624cd978c, infoPort=48162, 
infoSecurePort=0, ipcPort=33212, 
storageInfo=lv=-56;cid=testClusterID;nsid=2101821981;c=0), blocks: 0, 
hasStaleStorage: true, processing time: 86 msecs
   [junit4]   2> 819712 INFO  (IPC Server handler 3 on 59517) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-394f120c-9225-45db-9d4d-a79e0357d0a0 node 
DatanodeRegistration(127.0.0.1:38537, 
datanodeUuid=6109fe6a-418b-4810-8965-110624cd978c, infoPort=48162, 
infoSecurePort=0, ipcPort=33212, 
storageInfo=lv=-56;cid=testClusterID;nsid=2101821981;c=0), blocks: 0, 
hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 819976 INFO  
(TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [    ] o.a.s.c.ZkTestServer 
STARTING ZK TEST SERVER
   [junit4]   2> 819977 INFO  (Thread-1303) [    ] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 819977 INFO  (Thread-1303) [    ] o.a.s.c.ZkTestServer 
Starting server
   [junit4]   2> 820180 INFO  
(TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [    ] o.a.s.c.ZkTestServer 
start zk server on port:43044
   [junit4]   2> 820180 INFO  
(TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 820189 INFO  
(TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 820201 INFO  (zkCallback-698-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@6ea7009c 
name:ZooKeeperConnection Watcher:127.0.0.1:43044 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 820201 INFO  
(TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 820201 INFO  
(TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 820201 INFO  
(TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 820210 INFO  
(TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 820215 INFO  
(TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 820239 INFO  (zkCallback-699-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@1095833c 
name:ZooKeeperConnection Watcher:127.0.0.1:43044/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 820241 INFO  
(TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 820241 INFO  
(TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 820241 INFO  
(TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 820242 INFO  
(TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 820244 INFO  
(TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 820244 INFO  
(TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 820250 INFO  
(TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
 to /configs/conf1/solrconfig.xml
   [junit4]   2> 820250 INFO  
(TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 820252 INFO  
(TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/schema.xml
 to /configs/conf1/schema.xml
   [junit4]   2> 820252 INFO  
(TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 820253 INFO  
(TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
 to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 820253 INFO  
(TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [    ] 
o.a.s.c.c.SolrZkClient makePath: 
/configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 820255 INFO  
(TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
 to /configs/conf1/stopwords.txt
   [junit4]   2> 820255 INFO  
(TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 820262 INFO  
(TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt
 to /configs/conf1/protwords.txt
   [junit4]   2> 820262 INFO  
(TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 820263 INFO  
(TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml
 to /configs/conf1/currency.xml
   [junit4]   2> 820263 INFO  
(TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 820264 INFO  
(TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
 to /configs/conf1/enumsConfig.xml
   [junit4]   2> 820264 INFO  
(TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 820270 INFO  
(TEST-StressHdfsTest.test-seed#[66DE89DEA79A3E32]) [    ] 
o.a.s.c.AbstractZkTestCase put /x1/jenki

[...truncated too long message...]

ctory Closing directory, CoreContainer#isShutdown=true
   [junit4]   2> 4696298 INFO  (coreCloseExecutor-5736-thread-1) [    
x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedFacetPivotLongTailTest_66DE89DEA79A3E32-001/tempDir-001/shard2/cores/collection1/data/index
   [junit4]   2> 4696305 INFO  
(TEST-DistributedFacetPivotLongTailTest.test-seed#[66DE89DEA79A3E32]) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@17e66fe8{/_/d,null,UNAVAILABLE}
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 
'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=DistributedFacetPivotLongTailTest -Dtests.method=test 
-Dtests.seed=66DE89DEA79A3E32 -Dtests.multiplier=2 -Dtests.nightly=true 
-Dtests.slow=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt
 -Dtests.locale=cs -Dtests.timezone=Etc/GMT-6 -Dtests.asserts=true 
-Dtests.file.encoding=US-ASCII
   [junit4] ERROR    131s J0 | DistributedFacetPivotLongTailTest.test <<<
   [junit4]    > Throwable #1: 
org.apache.solr.client.solrj.SolrServerException: IOException occured when 
talking to server at: https://127.0.0.1:46729/_/d/collection1
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([66DE89DEA79A3E32:EE8AB604096653CA]:0)
   [junit4]    >        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:622)
   [junit4]    >        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
   [junit4]    >        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
   [junit4]    >        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
   [junit4]    >        at 
org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173)
   [junit4]    >        at 
org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138)
   [junit4]    >        at 
org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:152)
   [junit4]    >        at 
org.apache.solr.handler.component.DistributedFacetPivotLongTailTest.test(DistributedFacetPivotLongTailTest.java:71)
   [junit4]    >        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
   [junit4]    >        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
   [junit4]    >        at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: org.apache.http.conn.ConnectTimeoutException: 
Connect to 127.0.0.1:46729 [/127.0.0.1] failed: Read timed out
   [junit4]    >        at 
org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:143)
   [junit4]    >        at 
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
   [junit4]    >        at 
org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
   [junit4]    >        at 
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
   [junit4]    >        at 
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
   [junit4]    >        at 
org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
   [junit4]    >        at 
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
   [junit4]    >        at 
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
   [junit4]    >        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]    >        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
   [junit4]    >        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:513)
   [junit4]    >        ... 48 more
   [junit4]    > Caused by: java.net.SocketTimeoutException: Read timed out
   [junit4]    >        at java.net.SocketInputStream.socketRead0(Native Method)
   [junit4]    >        at 
java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
   [junit4]    >        at 
java.net.SocketInputStream.read(SocketInputStream.java:170)
   [junit4]    >        at 
java.net.SocketInputStream.read(SocketInputStream.java:141)
   [junit4]    >        at 
sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
   [junit4]    >        at 
sun.security.ssl.InputRecord.read(InputRecord.java:503)
   [junit4]    >        at 
sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
   [junit4]    >        at 
sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
   [junit4]    >        at 
sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
   [junit4]    >        at 
sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
   [junit4]    >        at 
org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394)
   [junit4]    >        at 
org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353)
   [junit4]    >        at 
org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
   [junit4]    >        ... 58 more
   [junit4]   2> 4696316 INFO  
(SUITE-DistributedFacetPivotLongTailTest-seed#[66DE89DEA79A3E32]-worker) [    ] 
o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedFacetPivotLongTailTest_66DE89DEA79A3E32-001
   [junit4]   2> NOTE: test params are: codec=Lucene62, sim=ClassicSimilarity, 
locale=cs, timezone=Etc/GMT-6
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 
1.8.0_102 (64-bit)/cpus=4,threads=1,free=251613816,total=477626368
   [junit4]   2> NOTE: All tests run in this JVM: 
[OverseerModifyCollectionTest, RequiredFieldsTest, 
DistributedDebugComponentTest, TestMissingGroups, TestIBSimilarityFactory, 
CdcrBootstrapTest, TestManagedSynonymFilterFactory, 
LeaderInitiatedRecoveryOnShardRestartTest, TestSubQueryTransformerCrossCore, 
TestOnReconnectListenerSupport, ZkStateReaderTest, TestSolrQueryParserResource, 
ResourceLoaderTest, TestConfigSetImmutable, TestSolr4Spatial, 
TestFreeTextSuggestions, TestReplicationHandlerBackup, ReturnFieldsTest, 
TestClassicSimilarityFactory, HdfsThreadLeakTest, 
SharedFSAutoReplicaFailoverUtilsTest, TestCSVLoader, 
LeaderElectionIntegrationTest, ZkNodePropsTest, TestRemoteStreaming, 
TestSchemaManager, EnumFieldTest, BasicZkTest, DirectoryFactoryTest, 
RemoteQueryErrorTest, SOLR749Test, TestLMDirichletSimilarityFactory, 
TestStressUserVersions, TestFieldCacheSortRandom, TestDistribDocBasedVersion, 
ChangedSchemaMergeTest, RuleEngineTest, TestPhraseSuggestions, 
TestCloudPseudoReturnFields, TestReplicationHandler, DisMaxRequestHandlerTest, 
TestLegacyFieldCache, TestComplexPhraseQParserPlugin, BasicDistributedZkTest, 
TestLRUCache, TestDistributedSearch, RestartWhileUpdatingTest, 
TestMergePolicyConfig, TestRebalanceLeaders, TestInitParams, CacheHeaderTest, 
BitVectorTest, TestArbitraryIndexDir, TestCloudDeleteByQuery, 
PathHierarchyTokenizerFactoryTest, TestSchemaNameResource, TestOrdValues, 
DistributedFacetPivotLargeTest, BlockJoinFacetDistribTest, TestFaceting, 
SynonymTokenizerTest, SpatialHeatmapFacetsTest, TestMiniSolrCloudClusterBase, 
OutOfBoxZkACLAndCredentialsProvidersTest, QueryEqualityTest, IndexSchemaTest, 
TestBadConfig, TestAddFieldRealTimeGet, BlockDirectoryTest, 
DefaultValueUpdateProcessorTest, TestCollationFieldDocValues, 
TestFoldingMultitermQuery, TestBlobHandler, SolrCloudExampleTest, 
DistributedSpellCheckComponentTest, ConnectionReuseTest, 
PingRequestHandlerTest, DocumentAnalysisRequestHandlerTest, 
AddSchemaFieldsUpdateProcessorFactoryTest, HdfsCollectionsAPIDistributedZkTest, 
TestRTGBase, CloneFieldUpdateProcessorFactoryTest, 
TestSizeLimitedDistributedMap, TestLuceneMatchVersion, TestRawResponseWriter, 
SignatureUpdateProcessorFactoryTest, TestXmlQParserPlugin, TestComponentsName, 
TestSQLHandler, HdfsLockFactoryTest, TestIntervalFaceting, 
TestCloudManagedSchema, TestDynamicFieldCollectionResource, 
TestSortingResponseWriter, TestHighlightDedupGrouping, UUIDFieldTest, 
ConnectionManagerTest, AnalysisAfterCoreReloadTest, TestSchemaVersionResource, 
JSONWriterTest, TestUninvertingReader, TermVectorComponentDistributedTest, 
TestGroupingSearch, SimpleCollectionCreateDeleteTest, 
DistributedFacetPivotLongTailTest]
   [junit4] Completed [632/633 (5!)] on J0 in 130.67s, 1 test, 1 error <<< 
FAILURES!

[...truncated 1 lines...]
   [junit4] JVM J0: stdout was not empty, see: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/temp/junit4-J0-20160907_083928_132.sysout
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Dumping heap to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/heapdumps/java_pid30315.hprof
 ...
   [junit4] Heap dump file created [707246725 bytes in 19.813 secs]
   [junit4] <<< JVM J0: EOF ----

[...truncated 11039 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:771:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:715:
 Some of the tests produced a heap dump, but did not fail. Maybe a suppressed 
OutOfMemoryError? Dumps created:
* java_pid30315.hprof

Total time: 367 minutes 35 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to