Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1385/

9 tests failed.
FAILED:  org.apache.lucene.spatial3d.TestGeo3DPoint.testRandomBig

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([F1997566B886F3EA]:0)


FAILED:  junit.framework.TestSuite.org.apache.lucene.spatial3d.TestGeo3DPoint

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([F1997566B886F3EA]:0)


FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest.test

Error Message:
The Monkey ran for over 45 seconds and no jetties were stopped - this is worth 
investigating!

Stack Trace:
java.lang.AssertionError: The Monkey ran for over 45 seconds and no jetties 
were stopped - this is worth investigating!
        at 
__randomizedtesting.SeedInfo.seed([AA9C3E941E8BCF89:22C8014EB077A271]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:587)
        at 
org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest.test(ChaosMonkeySafeLeaderWithPullReplicasTest.java:174)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest

Error Message:
8 threads leaked from SUITE scope at 
org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest:     1) 
Thread[id=17848, name=zkCallback-1820-thread-2, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
        at java.lang.Thread.run(Thread.java:748)    2) Thread[id=17738, 
name=StoppableCommitThread-EventThread, state=WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)    
3) Thread[id=17672, name=StoppableIndexingThread, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
    4) Thread[id=17673, name=StoppableIndexingThread, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
    5) Thread[id=17674, name=StoppableCommitThread, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:55)  
  6) Thread[id=17739, name=zkCallback-1820-thread-1, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
        at java.lang.Thread.run(Thread.java:748)    7) Thread[id=17849, 
name=zkCallback-1820-thread-3, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
        at java.lang.Thread.run(Thread.java:748)    8) Thread[id=17737, 
name=StoppableCommitThread-SendThread(127.0.0.1:35705), state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
         at 
org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)    
     at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 8 threads leaked from SUITE 
scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest: 
   1) Thread[id=17848, name=zkCallback-1820-thread-2, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=17738, name=StoppableCommitThread-EventThread, state=WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
   3) Thread[id=17672, name=StoppableIndexingThread, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   4) Thread[id=17673, name=StoppableIndexingThread, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   5) Thread[id=17674, name=StoppableCommitThread, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:55)
   6) Thread[id=17739, name=zkCallback-1820-thread-1, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   7) Thread[id=17849, name=zkCallback-1820-thread-3, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   8) Thread[id=17737, name=StoppableCommitThread-SendThread(127.0.0.1:35705), 
state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at 
org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
        at __randomizedtesting.SeedInfo.seed([AA9C3E941E8BCF89]:0)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) 
Thread[id=17848, name=zkCallback-1820-thread-2, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
        at java.lang.Thread.run(Thread.java:748)    2) Thread[id=17672, 
name=StoppableIndexingThread, state=BLOCKED, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at 
sun.nio.cs.UTF_8$Encoder.encodeLoop(UTF_8.java:691)         at 
java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:579)         at 
sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:271)         at 
sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)         at 
sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)         at 
java.io.OutputStreamWriter.write(OutputStreamWriter.java:220)         at 
java.io.Writer.write(Writer.java:157)         at 
com.carrotsearch.ant.tasks.junit4.gson.stream.JsonWriter.string(JsonWriter.java:574)
         at 
com.carrotsearch.ant.tasks.junit4.gson.stream.JsonWriter.writeDeferredName(JsonWriter.java:398)
         at 
com.carrotsearch.ant.tasks.junit4.gson.stream.JsonWriter.value(JsonWriter.java:413)
         at 
com.carrotsearch.ant.tasks.junit4.events.AbstractEvent.writeBinaryProperty(AbstractEvent.java:36)
         at 
com.carrotsearch.ant.tasks.junit4.events.AppendStdErrEvent.serialize(AppendStdErrEvent.java:30)
         at 
com.carrotsearch.ant.tasks.junit4.events.Serializer$2.run(Serializer.java:119)  
       at 
com.carrotsearch.ant.tasks.junit4.events.Serializer$2.run(Serializer.java:114)  
       at java.security.AccessController.doPrivileged(Native Method)         at 
com.carrotsearch.ant.tasks.junit4.events.Serializer.flushQueue(Serializer.java:114)
         at 
com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:99)
         at 
com.carrotsearch.ant.tasks.junit4.slave.SlaveMain$3$2.write(SlaveMain.java:472) 
        at 
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)         
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)         at 
java.io.PrintStream.flush(PrintStream.java:338)         at 
java.io.FilterOutputStream.flush(FilterOutputStream.java:140)         at 
java.io.PrintStream.write(PrintStream.java:482)         at 
sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)         at 
sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)         at 
sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)         at 
java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)         at 
java.io.PrintStream.write(PrintStream.java:527)         at 
java.io.PrintStream.print(PrintStream.java:669)         at 
java.io.PrintStream.println(PrintStream.java:823)         at 
java.lang.Throwable$WrappedPrintStream.println(Throwable.java:748)         at 
java.lang.Throwable.printStackTrace(Throwable.java:658)         at 
java.lang.Throwable.printStackTrace(Throwable.java:643)         at 
java.lang.Throwable.printStackTrace(Throwable.java:634)         at 
org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:121)
    3) Thread[id=17673, name=StoppableIndexingThread, state=RUNNABLE, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at 
org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:612)
         at 
org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)  
       at 
org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)        
 at 
org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)   
      at 
org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)  
       at 
org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)        
 at 
org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)   
      at 
org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)  
       at 
org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)        
 at 
org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)  
       at 
org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)  
       at 
org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)        
 at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)     
    at 
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)
         at 
org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)
         at 
org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)
         at 
org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)
         at 
org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:382)
         at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:251)
         at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
         at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)         
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)       
  at 
org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)
         at 
org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)
    4) Thread[id=17739, name=zkCallback-1820-thread-1, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
        at java.lang.Thread.run(Thread.java:748)    5) Thread[id=17849, 
name=zkCallback-1820-thread-3, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
        at java.lang.Thread.run(Thread.java:748)    6) Thread[id=17737, 
name=StoppableCommitThread-SendThread(127.0.0.1:35705), state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
         at 
org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)    
     at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie 
threads that couldn't be terminated:
   1) Thread[id=17848, name=zkCallback-1820-thread-2, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=17672, name=StoppableIndexingThread, state=BLOCKED, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.nio.cs.UTF_8$Encoder.encodeLoop(UTF_8.java:691)
        at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:579)
        at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:271)
        at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
        at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)
        at java.io.OutputStreamWriter.write(OutputStreamWriter.java:220)
        at java.io.Writer.write(Writer.java:157)
        at 
com.carrotsearch.ant.tasks.junit4.gson.stream.JsonWriter.string(JsonWriter.java:574)
        at 
com.carrotsearch.ant.tasks.junit4.gson.stream.JsonWriter.writeDeferredName(JsonWriter.java:398)
        at 
com.carrotsearch.ant.tasks.junit4.gson.stream.JsonWriter.value(JsonWriter.java:413)
        at 
com.carrotsearch.ant.tasks.junit4.events.AbstractEvent.writeBinaryProperty(AbstractEvent.java:36)
        at 
com.carrotsearch.ant.tasks.junit4.events.AppendStdErrEvent.serialize(AppendStdErrEvent.java:30)
        at 
com.carrotsearch.ant.tasks.junit4.events.Serializer$2.run(Serializer.java:119)
        at 
com.carrotsearch.ant.tasks.junit4.events.Serializer$2.run(Serializer.java:114)
        at java.security.AccessController.doPrivileged(Native Method)
        at 
com.carrotsearch.ant.tasks.junit4.events.Serializer.flushQueue(Serializer.java:114)
        at 
com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:99)
        at 
com.carrotsearch.ant.tasks.junit4.slave.SlaveMain$3$2.write(SlaveMain.java:472)
        at 
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
        at java.io.PrintStream.flush(PrintStream.java:338)
        at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)
        at java.io.PrintStream.write(PrintStream.java:482)
        at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
        at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
        at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)
        at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)
        at java.io.PrintStream.write(PrintStream.java:527)
        at java.io.PrintStream.print(PrintStream.java:669)
        at java.io.PrintStream.println(PrintStream.java:823)
        at java.lang.Throwable$WrappedPrintStream.println(Throwable.java:748)
        at java.lang.Throwable.printStackTrace(Throwable.java:658)
        at java.lang.Throwable.printStackTrace(Throwable.java:643)
        at java.lang.Throwable.printStackTrace(Throwable.java:634)
        at 
org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:121)
   3) Thread[id=17673, name=StoppableIndexingThread, state=RUNNABLE, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at 
org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:612)
        at 
org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)
        at 
org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at 
org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)
        at 
org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)
        at 
org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at 
org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)
        at 
org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)
        at 
org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at 
org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)
        at 
org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)
        at 
org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at 
org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)
        at 
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)
        at 
org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)
        at 
org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)
        at 
org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:382)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:251)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
        at 
org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)
        at 
org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)
   4) Thread[id=17739, name=zkCallback-1820-thread-1, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=17849, name=zkCallback-1820-thread-3, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=17737, name=StoppableCommitThread-SendThread(127.0.0.1:35705), 
state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at 
org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
        at __randomizedtesting.SeedInfo.seed([AA9C3E941E8BCF89]:0)


FAILED:  org.apache.solr.uninverting.TestDocTermOrds.testTriggerUnInvertLimit

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at 
__randomizedtesting.SeedInfo.seed([AA9C3E941E8BCF89:992E1650133C153E]:0)
        at 
org.apache.lucene.codecs.memory.DirectPostingsFormat$DirectField.<init>(DirectPostingsFormat.java:458)
        at 
org.apache.lucene.codecs.memory.DirectPostingsFormat$DirectFields.<init>(DirectPostingsFormat.java:129)
        at 
org.apache.lucene.codecs.memory.DirectPostingsFormat.fieldsProducer(DirectPostingsFormat.java:113)
        at 
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:292)
        at 
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:372)
        at 
org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:112)
        at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:78)
        at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:702)
        at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:300)
        at 
org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:865)
        at 
org.apache.solr.uninverting.TestDocTermOrds.testTriggerUnInvertLimit(TestDocTermOrds.java:188)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)


FAILED:  
org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testParallelExecutorStream

Error Message:
Error from server at http://127.0.0.1:43637/solr/workQueue_shard2_replica_n3: 
Expected mime type application/octet-stream but got text/html. <html> <head> 
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> 
<title>Error 404 </title> </head> <body> <h2>HTTP ERROR: 404</h2> <p>Problem 
accessing /solr/workQueue_shard2_replica_n3/update. Reason: <pre>    Can not 
find: /solr/workQueue_shard2_replica_n3/update</pre></p> <hr /><a 
href="http://eclipse.org/jetty";>Powered by Jetty:// 9.3.20.v20170531</a><hr/> 
</body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from 
server at http://127.0.0.1:43637/solr/workQueue_shard2_replica_n3: Expected 
mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 404 </title>
</head>
<body>
<h2>HTTP ERROR: 404</h2>
<p>Problem accessing /solr/workQueue_shard2_replica_n3/update. Reason:
<pre>    Can not find: /solr/workQueue_shard2_replica_n3/update</pre></p>
<hr /><a href="http://eclipse.org/jetty";>Powered by Jetty:// 
9.3.20.v20170531</a><hr/>
</body>
</html>

        at 
__randomizedtesting.SeedInfo.seed([DA32B9270A1D897C:6725CC3E3331B421]:0)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:539)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:993)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:862)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at 
org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:233)
        at 
org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testParallelExecutorStream(StreamExpressionTest.java:7211)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: 
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at http://127.0.0.1:43637/solr/workQueue_shard2_replica_n3: 
Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 404 </title>
</head>
<body>
<h2>HTTP ERROR: 404</h2>
<p>Problem accessing /solr/workQueue_shard2_replica_n3/update. Reason:
<pre>    Can not find: /solr/workQueue_shard2_replica_n3/update</pre></p>
<hr /><a href="http://eclipse.org/jetty";>Powered by Jetty:// 
9.3.20.v20170531</a><hr/>
</body>
</html>

        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:591)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:516)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        ... 1 more


FAILED:  org.apache.solr.cloud.TestSegmentSorting.testSegmentTerminateEarly

Error Message:
responseHeader.segmentTerminatedEarly missing/false in 
{zkConnected=true,segmentTerminatedEarly=false,status=0,QTime=31}

Stack Trace:
java.lang.AssertionError: responseHeader.segmentTerminatedEarly missing/false 
in {zkConnected=true,segmentTerminatedEarly=false,status=0,QTime=31}
        at 
__randomizedtesting.SeedInfo.seed([AA9C3E941E8BCF89:7A3AF939C2D95BE3]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at 
org.apache.solr.cloud.SegmentTerminateEarlyTestState.queryTimestampDescendingSegmentTerminateEarlyYes(SegmentTerminateEarlyTestState.java:150)
        at 
org.apache.solr.cloud.TestSegmentSorting.testSegmentTerminateEarly(TestSegmentSorting.java:109)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test

Error Message:
No live SolrServers available to handle this 
request:[http://127.0.0.1:45965/y_ot/solrj_collection4, 
http://127.0.0.1:43317/y_ot/solrj_collection4, 
http://127.0.0.1:44144/y_ot/solrj_collection4]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available 
to handle this request:[http://127.0.0.1:45965/y_ot/solrj_collection4, 
http://127.0.0.1:43317/y_ot/solrj_collection4, 
http://127.0.0.1:44144/y_ot/solrj_collection4]
        at 
__randomizedtesting.SeedInfo.seed([AA9C3E941E8BCF89:22C8014EB077A271]:0)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:462)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1121)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:862)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:967)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:957)
        at 
org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.queryAndAssertResultSize(SharedFSAutoReplicaFailoverTest.java:342)
        at 
org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.testBasics(SharedFSAutoReplicaFailoverTest.java:253)
        at 
org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test(SharedFSAutoReplicaFailoverTest.java:136)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: 
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at http://127.0.0.1:43317/y_ot/solrj_collection4: no servers 
hosting shard: shard2
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:627)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        ... 51 more




Build Log:
[...truncated 10445 lines...]
   [junit4] Suite: org.apache.lucene.spatial3d.TestGeo3DPoint
   [junit4]   2> sept. 11, 2017 5:47:09 AM 
com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
   [junit4]   2> WARNING: Suite execution timed out: 
org.apache.lucene.spatial3d.TestGeo3DPoint
   [junit4]   2>    1) Thread[id=9, name=JUnit4-serializer-daemon, 
state=TIMED_WAITING, group=main]
   [junit4]   2>         at java.lang.Thread.sleep(Native Method)
   [junit4]   2>         at 
com.carrotsearch.ant.tasks.junit4.events.Serializer$1.run(Serializer.java:50)
   [junit4]   2>    2) Thread[id=1, name=main, state=WAITING, group=main]
   [junit4]   2>         at java.lang.Object.wait(Native Method)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1252)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1326)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:636)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:493)
   [junit4]   2>         at 
com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:251)
   [junit4]   2>         at 
com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:368)
   [junit4]   2>         at 
com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
   [junit4]   2>    3) Thread[id=12, 
name=TEST-TestGeo3DPoint.testRandomBig-seed#[F1997566B886F3EA], state=RUNNABLE, 
group=TGRP-TestGeo3DPoint]
   [junit4]   2>         at 
org.apache.lucene.spatial3d.geom.Plane.findIntersections(Plane.java:650)
   [junit4]   2>         at 
org.apache.lucene.spatial3d.geom.GeoComplexPolygon.isWithin(GeoComplexPolygon.java:204)
   [junit4]   2>         at 
org.apache.lucene.spatial3d.geom.GeoBaseMembershipShape.isWithin(GeoBaseMembershipShape.java:36)
   [junit4]   2>         at 
org.apache.lucene.spatial3d.geom.GeoBaseShape.getBounds(GeoBaseShape.java:35)
   [junit4]   2>         at 
org.apache.lucene.spatial3d.geom.GeoComplexPolygon.getBounds(GeoComplexPolygon.java:441)
   [junit4]   2>         at 
org.apache.lucene.spatial3d.TestGeo3DPoint.verify(TestGeo3DPoint.java:846)
   [junit4]   2>         at 
org.apache.lucene.spatial3d.TestGeo3DPoint.doTestRandom(TestGeo3DPoint.java:548)
   [junit4]   2>         at 
org.apache.lucene.spatial3d.TestGeo3DPoint.testRandomBig(TestGeo3DPoint.java:480)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
   [junit4]   2>         at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
   [junit4]   2>         at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>    4) Thread[id=11, 
name=SUITE-TestGeo3DPoint-seed#[F1997566B886F3EA], state=RUNNABLE, 
group=TGRP-TestGeo3DPoint]
   [junit4]   2>         at java.lang.Thread.getStackTrace(Thread.java:1559)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:696)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:693)
   [junit4]   2>         at java.security.AccessController.doPrivileged(Native 
Method)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.getStackTrace(ThreadLeakControl.java:693)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.getThreadsWithTraces(ThreadLeakControl.java:709)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:689)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:65)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:415)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:705)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:139)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:626)
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 
'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestGeo3DPoint 
-Dtests.method=testRandomBig -Dtests.seed=F1997566B886F3EA -Dtests.multiplier=2 
-Dtests.nightly=true -Dtests.slow=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt
 -Dtests.locale=lv -Dtests.timezone=Asia/Jayapura -Dtests.asserts=true 
-Dtests.file.encoding=US-ASCII
   [junit4] ERROR   6759s J1 | TestGeo3DPoint.testRandomBig <<<
   [junit4]    > Throwable #1: java.lang.Exception: Test abandoned because 
suite timeout was reached.
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([F1997566B886F3EA]:0)
   [junit4]   2> sept. 11, 2017 5:47:09 AM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked 
thread(s).
   [junit4]   2> sept. 11, 2017 5:47:30 AM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at 
org.apache.lucene.spatial3d.TestGeo3DPoint: 
   [junit4]   2>    1) Thread[id=12, 
name=TEST-TestGeo3DPoint.testRandomBig-seed#[F1997566B886F3EA], state=RUNNABLE, 
group=TGRP-TestGeo3DPoint]
   [junit4]   2>         at 
org.apache.lucene.spatial3d.geom.Plane.findIntersectionBounds(Plane.java:1081)
   [junit4]   2>         at 
org.apache.lucene.spatial3d.geom.Plane.recordBounds(Plane.java:1214)
   [junit4]   2>         at 
org.apache.lucene.spatial3d.geom.XYZBounds.addIntersection(XYZBounds.java:262)
   [junit4]   2>         at 
org.apache.lucene.spatial3d.geom.GeoConvexPolygon.getBounds(GeoConvexPolygon.java:486)
   [junit4]   2>         at 
org.apache.lucene.spatial3d.geom.GeoBaseCompositeShape.getBounds(GeoBaseCompositeShape.java:131)
   [junit4]   2>         at 
org.apache.lucene.spatial3d.TestGeo3DPoint.verify(TestGeo3DPoint.java:846)
   [junit4]   2>         at 
org.apache.lucene.spatial3d.TestGeo3DPoint.doTestRandom(TestGeo3DPoint.java:548)
   [junit4]   2>         at 
org.apache.lucene.spatial3d.TestGeo3DPoint.testRandomBig(TestGeo3DPoint.java:480)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
   [junit4]   2>         at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
   [junit4]   2>         at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> sept. 11, 2017 5:47:30 AM 
com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=12, 
name=TEST-TestGeo3DPoint.testRandomBig-seed#[F1997566B886F3EA], state=RUNNABLE, 
group=TGRP-TestGeo3DPoint]
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/spatial3d/test/J1/temp/lucene.spatial3d.TestGeo3DPoint_F1997566B886F3EA-001
   [junit4]   2> Sep 10, 2017 8:47:30 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: All leaked threads terminated.
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70), 
sim=RandomSimilarity(queryNorm=true): {}, locale=lv, timezone=Asia/Jayapura
   [junit4]   2> NOTE: Linux 4.4.0-83-generic amd64/Oracle Corporation 
1.8.0_144 (64-bit)/cpus=4,threads=1,free=209486408,total=440926208
   [junit4]   2> NOTE: All tests run in this JVM: [TestGeo3DPoint]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 
'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestGeo3DPoint 
-Dtests.seed=F1997566B886F3EA -Dtests.multiplier=2 -Dtests.nightly=true 
-Dtests.slow=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt
 -Dtests.locale=lv -Dtests.timezone=Asia/Jayapura -Dtests.asserts=true 
-Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J1 | TestGeo3DPoint (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 
7200000 msec).
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([F1997566B886F3EA]:0)
   [junit4] Completed [13/15 (1!)] on J1 in 7221.93s, 6 tests, 2 errors <<< 
FAILURES!

[...truncated 1996 lines...]
   [junit4] Suite: org.apache.solr.uninverting.TestDocTermOrds
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 
'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDocTermOrds 
-Dtests.method=testTriggerUnInvertLimit -Dtests.seed=AA9C3E941E8BCF89 
-Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt
 -Dtests.locale=ar-MA -Dtests.timezone=Asia/Tehran -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8
   [junit4] ERROR    503s J2 | TestDocTermOrds.testTriggerUnInvertLimit <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: GC overhead limit 
exceeded
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([AA9C3E941E8BCF89:992E1650133C153E]:0)
   [junit4]    >        at 
org.apache.lucene.codecs.memory.DirectPostingsFormat$DirectField.<init>(DirectPostingsFormat.java:458)
   [junit4]    >        at 
org.apache.lucene.codecs.memory.DirectPostingsFormat$DirectFields.<init>(DirectPostingsFormat.java:129)
   [junit4]    >        at 
org.apache.lucene.codecs.memory.DirectPostingsFormat.fieldsProducer(DirectPostingsFormat.java:113)
   [junit4]    >        at 
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:292)
   [junit4]    >        at 
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:372)
   [junit4]    >        at 
org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:112)
   [junit4]    >        at 
org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:78)
   [junit4]    >        at 
org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:702)
   [junit4]    >        at 
org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:300)
   [junit4]    >        at 
org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:865)
   [junit4]    >        at 
org.apache.solr.uninverting.TestDocTermOrds.testTriggerUnInvertLimit(TestDocTermOrds.java:188)
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.uninverting.TestDocTermOrds_AA9C3E941E8BCF89-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): 
{field=PostingsFormat(name=Direct), foo=PostingsFormat(name=Direct), 
id=BlockTreeOrds(blocksize=128)}, docValues:{}, maxPointsInLeafNode=1298, 
maxMBSortInHeap=5.151702879083166, sim=RandomSimilarity(queryNorm=false): 
{field=DFR I(ne)BZ(0.3)}, locale=ar-MA, timezone=Asia/Tehran
   [junit4]   2> NOTE: Linux 4.4.0-83-generic amd64/Oracle Corporation 
1.8.0_144 (64-bit)/cpus=4,threads=1,free=456275808,total=477626368
   [junit4]   2> NOTE: All tests run in this JVM: [TestStressInPlaceUpdates, 
TestFoldingMultitermQuery, CdcrBootstrapTest, DirectUpdateHandlerOptimizeTest, 
TestNumericRangeQuery32, TestExactStatsCache, TestCoreAdminApis, 
BigEndianAscendingWordDeserializerTest, SimpleMLTQParserTest, TestDocTermOrds]
   [junit4] Completed [62/732 (1!)] on J2 in 510.92s, 10 tests, 1 error <<< 
FAILURES!

[...truncated 118 lines...]
   [junit4] Suite: org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest
   [junit4]   2> 1743348 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[AA9C3E941E8BCF89]-worker) [    ] 
o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: 
test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_AA9C3E941E8BCF89-001/init-core-data-001
   [junit4]   2> 1743349 WARN  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[AA9C3E941E8BCF89]-worker) [    ] 
o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
   [junit4]   2> 1743362 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[AA9C3E941E8BCF89]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) 
w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 1743363 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[AA9C3E941E8BCF89]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: 
@org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=None)
   [junit4]   2> 1743363 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[AA9C3E941E8BCF89]-worker) [    ] 
o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /y_ot/
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 1743522 WARN  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[AA9C3E941E8BCF89]-worker) [    ] 
o.a.h.m.i.MetricsConfig Cannot locate configuration: tried 
hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 1743681 WARN  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[AA9C3E941E8BCF89]-worker) [    ] 
o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1743683 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[AA9C3E941E8BCF89]-worker) [    ] 
o.m.log jetty-6.1.26
   [junit4]   2> 1743732 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[AA9C3E941E8BCF89]-worker) [    ] 
o.m.log Extract 
jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/hdfs
 to ./temp/Jetty_lucene2.us.west_apache_org_40282_hdfs____cc2al6/webapp
   [junit4]   2> 1744947 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[AA9C3E941E8BCF89]-worker) [    ] 
o.m.log Started 
HttpServer2$selectchannelconnectorwithsafestar...@lucene2-us-west.apache.org:40282
   [junit4]   2> 1745410 WARN  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[AA9C3E941E8BCF89]-worker) [    ] 
o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1745411 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[AA9C3E941E8BCF89]-worker) [    ] 
o.m.log jetty-6.1.26
   [junit4]   2> 1745492 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[AA9C3E941E8BCF89]-worker) [    ] 
o.m.log Extract 
jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode
 to ./temp/Jetty_localhost_36444_datanode____.d0y3hk/webapp
   [junit4]   2> 1746896 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[AA9C3E941E8BCF89]-worker) [    ] 
o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36444
   [junit4]   2> 1747190 WARN  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[AA9C3E941E8BCF89]-worker) [    ] 
o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1747191 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[AA9C3E941E8BCF89]-worker) [    ] 
o.m.log jetty-6.1.26
   [junit4]   2> 1747261 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[AA9C3E941E8BCF89]-worker) [    ] 
o.m.log Extract 
jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode
 to ./temp/Jetty_localhost_43735_datanode____.t2l3bz/webapp
   [junit4]   2> 1748137 ERROR (DataNode: 
[[[DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_AA9C3E941E8BCF89-001/tempDir-001/hdfsBaseDir/data/data1/,
 
[DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_AA9C3E941E8BCF89-001/tempDir-001/hdfsBaseDir/data/data2/]]
  heartbeating to lucene2-us-west.apache.org/127.0.0.1:32984) [    ] 
o.a.h.h.s.d.DirectoryScanner 
dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 
ms/sec. Assuming default value of 1000
   [junit4]   2> 1748173 INFO  (Block report processor) [    ] BlockStateChange 
BLOCK* processReport 0x126e6e0787fee0: from storage 
DS-fa64249a-8738-4bdc-8991-9ee6fa2c4666 node 
DatanodeRegistration(127.0.0.1:35939, 
datanodeUuid=9043c029-2501-438c-a135-1400d8334e28, infoPort=38818, 
infoSecurePort=0, ipcPort=38134, 
storageInfo=lv=-56;cid=testClusterID;nsid=1916623432;c=0), blocks: 0, 
hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 1748173 INFO  (Block report processor) [    ] BlockStateChange 
BLOCK* processReport 0x126e6e0787fee0: from storage 
DS-1b4f9525-af8d-4a4b-a092-c41bfdf1cce2 node 
DatanodeRegistration(127.0.0.1:35939, 
datanodeUuid=9043c029-2501-438c-a135-1400d8334e28, infoPort=38818, 
infoSecurePort=0, ipcPort=38134, 
storageInfo=lv=-56;cid=testClusterID;nsid=1916623432;c=0), blocks: 0, 
hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 1748340 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[AA9C3E941E8BCF89]-worker) [    ] 
o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43735
   [junit4]   2> 1749698 ERROR (DataNode: 
[[[DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_AA9C3E941E8BCF89-001/tempDir-001/hdfsBaseDir/data/data3/,
 
[DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_AA9C3E941E8BCF89-001/tempDir-001/hdfsBaseDir/data/data4/]]
  heartbeating to lucene2-us-west.apache.org/127.0.0.1:32984) [    ] 
o.a.h.h.s.d.DirectoryScanner 
dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 
ms/sec. Assuming default value of 1000
   [junit4]   2> 1749743 INFO  (Block report processor) [    ] BlockStateChange 
BLOCK* processReport 0x126e6e646dda62: from storage 
DS-75ba2639-bcdb-46b7-bae2-91494f40a253 node 
DatanodeRegistration(127.0.0.1:38033, 
datanodeUuid=8e66bd23-2ace-4998-83cd-0e77fa16bcca, infoPort=42202, 
infoSecurePort=0, ipcPort=35172, 
storageInfo=lv=-56;cid=testClusterID;nsid=1916623432;c=0), blocks: 0, 
hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 1749757 INFO  (Block report processor) [    ] BlockStateChange 
BLOCK* processReport 0x126e6e646dda62: from storage 
DS-4c6325fc-fb57-4941-8554-1831b0745097 node 
DatanodeRegistration(127.0.0.1:38033, 
datanodeUuid=8e66bd23-2ace-4998-83cd-0e77fa16bcca, infoPort=42202, 
infoSecurePort=0, ipcPort=35172, 
storageInfo=lv=-56;cid=testClusterID;nsid=1916623432;c=0), blocks: 0, 
hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 1750052 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[AA9C3E941E8BCF89]) [    ] 
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1750059 INFO  (Thread-1156) [    ] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1750059 INFO  (Thread-1156) [    ] o.a.s.c.ZkTestServer 
Starting server
   [junit4]   2> 1750159 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[AA9C3E941E8BCF89]) [    ] 
o.a.s.c.ZkTestServer start zk server on port:33673
   [junit4]   2> 1750228 ERROR (Thread-1156) [    ] o.a.z.s.ZooKeeperServer 
ZKShutdownHandler is not registered, so ZooKeeper server won't take any action 
on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1750333 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[AA9C3E941E8BCF89]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
 to /configs/conf1/solrconfig.xml
   [junit4]   2> 1750363 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[AA9C3E941E8BCF89]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml
 to /configs/conf1/schema.xml
   [junit4]   2> 1755755 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[AA9C3E941E8BCF89]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
 to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1755786 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[AA9C3E941E8BCF89]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
 to /configs/conf1/stopwords.txt
   [junit4]   2> 1755786 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[AA9C3E941E8BCF89]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt
 to /configs/conf1/protwords.txt
   [junit4]   2> 1755800 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[AA9C3E941E8BCF89]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml
 to /configs/conf1/currency.xml
   [junit4]   2> 1755801 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[AA9C3E941E8BCF89]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
 to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1755802 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[AA9C3E941E8BCF89]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
 to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1755802 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[AA9C3E941E8BCF89]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
 to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1755803 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[AA9C3E941E8BCF89]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
 to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1755804 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[AA9C3E941E8BCF89]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
 to /configs/conf1/synonyms.txt
   [junit4]   2> 1755855 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[AA9C3E941E8BCF89]) [    ] 
o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly 
asked otherwise
   [junit4]   2> 1756547 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[AA9C3E941E8BCF89]) [    ] 
o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 1756561 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[AA9C3E941E8BCF89]) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@d21a651{/y_ot,null,AVAILABLE}
   [junit4]   2> 1756562 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[AA9C3E941E8BCF89]) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@49c45020{HTTP/1.1,[http/1.1]}{127.0.0.1:40306}
   [junit4]   2> 1756562 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[AA9C3E941E8BCF89]) [    ] 
o.e.j.s.Server Started @1769167ms
   [junit4]   2> 1756562 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[AA9C3E941E8BCF89]) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {replicaType=NRT, 
hostContext=/y_ot, hostPort=40306, 
coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_AA9C3E941E8BCF89-001/control-001/cores}
   [junit4]   2> 1756562 ERROR 
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[AA9C3E941E8BCF89]) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 1756562 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[AA9C3E941E8BCF89]) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 
8.0.0
   [junit4]   2> 1756562 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[AA9C3E941E8BCF89]) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1756562 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[AA9C3E941E8BCF89]) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config 
dir: null
   [junit4]   2> 1756562 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[AA9C3E941E8BCF89]) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2017-09-10T21:24:27.544Z
   [junit4]   2> 1756670 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[AA9C3E941E8BCF89]) [    ] 
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 1756670 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[AA9C3E941E8BCF89]) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_AA9C3E941E8BCF89-001/control-001/solr.xml
   [junit4]   2> 1756704 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[AA9C3E941E8BCF89]) [    ] 
o.a.s.c.SolrXmlConfig MBean server found: 
com.sun.jmx.mbeanserver.JmxMBeanServer@e7f1407, but no JMX reporters were 
configured - adding default JMX reporter.
   [junit4]   2> 1756735 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[AA9C3E941E8BCF89]) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33673/solr
   [junit4]   2> 1757227 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[AA9C3E941E8BCF89]) 
[n:127.0.0.1:40306_y_ot    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1757227 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[AA9C3E941E8BCF89]) 
[n:127.0.0.1:40306_y_ot    ] o.a.s.c.OverseerElectionContext I am going to be 
the leader 127.0.0.1:40306_y_ot
   [junit4]   2> 1757228 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[AA9C3E941E8BCF89]) 
[n:127.0.0.1:40306_y_ot    ] o.a.s.c.Overseer Overseer 
(id=98636835132276740-127.0.0.1:40306_y_ot-n_0000000000) starting
   [junit4]   2> 1757271 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[AA9C3E941E8BCF89]) 
[n:127.0.0.1:40306_y_ot    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:40306_y_ot
   [junit4]   2> 1757340 INFO  
(zkCallback-527-thread-1-processing-n:127.0.0.1:40306_y_ot) 
[n:127.0.0.1:40306_y_ot    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 1757842 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[AA9C3E941E8BCF89]) 
[n:127.0.0.1:40306_y_ot    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@e7f1407
   [junit4]   2> 1758005 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[AA9C3E941E8BCF89]) 
[n:127.0.0.1:40306_y_ot    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@e7f1407
   [junit4]   2> 1758005 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[AA9C3E941E8BCF89]) 
[n:127.0.0.1:40306_y_ot    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@e7f1407
   [junit4]   2> 1758055 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[AA9C3E941E8BCF89]) 
[n:127.0.0.1:40306_y_ot    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_AA9C3E941E8BCF89-001/control-001/cores
   [junit4]   2> 1758453 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[AA9C3E941E8BCF89]) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1758498 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[AA9C3E941E8BCF89]) [    ] 
o.a.s.c.s.i.

[...truncated too long message...]

 (coreCloseExecutor-759-thread-2) [n:127.0.0.1:42692_solr c:destination 
s:shard2 r:core_node4 x:destination_shard2_replica_n2] 
o.a.s.m.SolrMetricManager Closing metric reporters for 
registry=solr.collection.destination.shard2.leader, tag=1789886435
   [junit4]   2> 606809 INFO  (coreCloseExecutor-757-thread-1) 
[n:127.0.0.1:33831_solr c:collection1 s:shard1 r:core_node3 
x:collection1_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric 
reporters for registry=solr.collection.collection1.shard1.leader, tag=1267855744
   [junit4]   2> 606810 INFO  (coreCloseExecutor-758-thread-1) 
[n:127.0.0.1:42467_solr c:mainCorpus s:shard1 r:core_node3 
x:mainCorpus_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric 
reporters for registry=solr.collection.mainCorpus.shard1.leader, tag=1461853015
   [junit4]   2> 606905 INFO  (coreCloseExecutor-760-thread-3) 
[n:127.0.0.1:43637_solr c:mainCorpus s:shard2 r:core_node4 
x:mainCorpus_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric 
reporters for registry=solr.collection.mainCorpus.shard2.leader, tag=957965063
   [junit4]   2> 606908 INFO  
(zkCallback-467-thread-1-processing-n:127.0.0.1:42467_solr) 
[n:127.0.0.1:42467_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (4) -> (3)
   [junit4]   2> 606942 INFO  (coreCloseExecutor-760-thread-2) 
[n:127.0.0.1:43637_solr c:workQueue s:shard2 r:core_node4 
x:workQueue_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric 
reporters for registry=solr.collection.workQueue.shard2.leader, tag=36656337
   [junit4]   2> 606954 INFO  (coreCloseExecutor-760-thread-4) 
[n:127.0.0.1:43637_solr c:destination s:shard1 r:core_node3 
x:destination_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric 
reporters for registry=solr.core.destination.shard1.replica_n1, tag=35381388
   [junit4]   2> 607015 INFO  (coreCloseExecutor-760-thread-4) 
[n:127.0.0.1:43637_solr c:destination s:shard1 r:core_node3 
x:destination_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@68949e32: rootName = 
solr_43637, domain = solr.core.destination.shard1.replica_n1, service url = 
null, agent id = null] for registry solr.core.destination.shard1.replica_n1 / 
com.codahale.metrics.MetricRegistry@eb14e4c
   [junit4]   2> 607043 INFO  
(zkCallback-468-thread-4-processing-n:127.0.0.1:43637_solr) 
[n:127.0.0.1:43637_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (4) -> (2)
   [junit4]   2> 607063 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from 
client sessionid 0x15e6dec567b000c, likely client has closed socket
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 607066 INFO  (coreCloseExecutor-760-thread-4) 
[n:127.0.0.1:43637_solr c:destination s:shard1 r:core_node3 
x:destination_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric 
reporters for registry=solr.collection.destination.shard1.leader, tag=35381388
   [junit4]   2> 607156 INFO  
(zkCallback-467-thread-1-processing-n:127.0.0.1:42467_solr) 
[n:127.0.0.1:42467_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (3) -> (2)
   [junit4]   2> 607205 INFO  (jetty-closer-452-thread-4) [    ] 
o.a.s.c.Overseer Overseer 
(id=98637103679340551-127.0.0.1:42467_solr-n_0000000000) closing
   [junit4]   2> 607205 INFO  
(OverseerStateUpdate-98637103679340551-127.0.0.1:42467_solr-n_0000000000) 
[n:127.0.0.1:42467_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 
127.0.0.1:42467_solr
   [junit4]   2> 607239 INFO  (jetty-closer-452-thread-4) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@4b625a05{/solr,null,UNAVAILABLE}
   [junit4]   2> 607239 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from 
client sessionid 0x15e6dec567b0007, likely client has closed socket
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 608407 WARN  
(zkCallback-469-thread-4-processing-n:127.0.0.1:33831_solr) 
[n:127.0.0.1:33831_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, 
but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 608407 INFO  (jetty-closer-452-thread-1) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@58868887{/solr,null,UNAVAILABLE}
   [junit4]   2> 608450 WARN  
(zkCallback-476-thread-3-processing-n:127.0.0.1:42692_solr) 
[n:127.0.0.1:42692_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, 
but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 608450 INFO  (jetty-closer-452-thread-2) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@6faf64f1{/solr,null,UNAVAILABLE}
   [junit4]   2> 608738 WARN  
(zkCallback-468-thread-4-processing-n:127.0.0.1:43637_solr) 
[n:127.0.0.1:43637_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, 
but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 608739 INFO  (jetty-closer-452-thread-3) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@a47aa07{/solr,null,UNAVAILABLE}
   [junit4]   2> 608741 ERROR 
(SUITE-StreamExpressionTest-seed#[DA32B9270A1D897C]-worker) [    ] 
o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper 
server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 608742 INFO  
(SUITE-StreamExpressionTest-seed#[DA32B9270A1D897C]-worker) [    ] 
o.a.s.c.ZkTestServer connecting to 127.0.0.1:45072 45072
   [junit4]   2> 608875 INFO  (Thread-152) [    ] o.a.s.c.ZkTestServer 
connecting to 127.0.0.1:45072 45072
   [junit4]   2> 608876 WARN  (Thread-152) [    ] o.a.s.c.ZkTestServer Watch 
limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2>        91      /solr/aliases.json
   [junit4]   2>        91      /solr/clusterprops.json
   [junit4]   2>        12      /solr/configs/conf
   [junit4]   2>        8       /solr/configs/ml
   [junit4]   2>        4       /solr/security.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2>        91      /solr/clusterstate.json
   [junit4]   2>        40      
/solr/collections/destinationCollection/state.json
   [junit4]   2>        32      
/solr/collections/parallelDestinationCollection1/state.json
   [junit4]   2>        24      /solr/collections/mainCorpus/state.json
   [junit4]   2>        23      
/solr/collections/parallelDestinationCollection/state.json
   [junit4]   2>        22      /solr/collections/destination/state.json
   [junit4]   2>        22      /solr/collections/workQueue/state.json
   [junit4]   2>        16      /solr/collections/collection1/state.json
   [junit4]   2>        10      /solr/collections/uknownCollection/state.json
   [junit4]   2>        10      /solr/collections/modelCollection/state.json
   [junit4]   2>        10      
/solr/collections/checkpointCollection/state.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2>        91      /solr/live_nodes
   [junit4]   2>        91      /solr/collections
   [junit4]   2> 
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-solrj/test/J2/temp/solr.client.solrj.io.stream.StreamExpressionTest_DA32B9270A1D897C-001
   [junit4]   2> Sep 10, 2017 10:36:30 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked 
thread(s).
   [junit4]   2> NOTE: test params are: codec=Lucene70, 
sim=RandomSimilarity(queryNorm=false): {}, locale=ar-KW, 
timezone=America/Anchorage
   [junit4]   2> NOTE: Linux 4.4.0-83-generic amd64/Oracle Corporation 
1.8.0_144 (64-bit)/cpus=4,threads=1,free=279420544,total=501219328
   [junit4]   2> NOTE: All tests run in this JVM: [TestZkConfigManager, 
SquareRootEvaluatorTest, AscEvaluatorTest, NaturalLogEvaluatorTest, 
SolrParamTest, RawValueEvaluatorTest, HttpSolrClientConPoolTest, StreamingTest, 
TestCoreAdmin, StreamExpressionParserTest, TestJavaBinCodec, 
CloudSolrClientCacheTest, HyperbolicTangentEvaluatorTest, 
ConcurrentUpdateSolrClientBuilderTest, SolrSchemalessExampleTest, 
SolrExceptionTest, TestBatchUpdate, NamedListTest, LessThanEvaluatorTest, 
TestConfigSetAdminRequest, JettyWebappTest, TestPathTrie, ShardParamsTest, 
SolrExampleStreamingBinaryTest, SolrExampleBinaryTest, SolrExampleJettyTest, 
SolrExampleEmbeddedTest, TestSolrProperties, MergeIndexesEmbeddedTest, 
TestDocumentObjectBinder, SolrQueryTest, TestHash, ModifiableSolrParamsTest, 
TestXMLEscaping, TestUpdateRequestCodec, ClientUtilsTest, 
AnlysisResponseBaseTest, GetByIdTest, GraphExpressionTest, StreamExpressionTest]
   [junit4] Completed [145/145 (1!)] on J2 in 232.62s, 108 tests, 1 error <<< 
FAILURES!

[...truncated 7287 lines...]
BUILD FAILED
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:834:
 The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:778:
 Some of the tests produced a heap dump, but did not fail. Maybe a suppressed 
OutOfMemoryError? Dumps created:
* java_pid10708.hprof

Total time: 634 minutes 22 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to