Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1334/

7 tests failed.
FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
8 threads leaked from SUITE scope at 
org.apache.solr.cloud.ChaosMonkeySafeLeaderTest:     1) Thread[id=48255, 
name=zkCallback-1500-thread-3, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)    2) Thread[id=48157, 
name=StoppableIndexingThread, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderTest]         at java.lang.Thread.sleep(Native 
Method)         at 
org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
    3) Thread[id=48174, 
name=StoppableIndexingThread-SendThread(127.0.0.1:55407), state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderTest]         at java.lang.Thread.sleep(Native 
Method)         at 
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)    4) 
Thread[id=48156, name=StoppableIndexingThread, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderTest]         at java.lang.Thread.sleep(Native 
Method)         at 
org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
    5) Thread[id=48176, name=zkCallback-1500-thread-1, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)    6) Thread[id=48256, 
name=zkCallback-1500-thread-4, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)    7) Thread[id=48254, 
name=zkCallback-1500-thread-2, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)    8) Thread[id=48175, 
name=StoppableIndexingThread-EventThread, state=WAITING, 
group=TGRP-ChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 8 threads leaked from SUITE 
scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest: 
   1) Thread[id=48255, name=zkCallback-1500-thread-3, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=48157, name=StoppableIndexingThread, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   3) Thread[id=48174, 
name=StoppableIndexingThread-SendThread(127.0.0.1:55407), state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)
   4) Thread[id=48156, name=StoppableIndexingThread, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   5) Thread[id=48176, name=zkCallback-1500-thread-1, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=48256, name=zkCallback-1500-thread-4, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   7) Thread[id=48254, name=zkCallback-1500-thread-2, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   8) Thread[id=48175, name=StoppableIndexingThread-EventThread, state=WAITING, 
group=TGRP-ChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
        at __randomizedtesting.SeedInfo.seed([304B34BAA0DA99AA]:0)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) 
Thread[id=48255, name=zkCallback-1500-thread-3, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)    2) Thread[id=48157, 
name=StoppableIndexingThread, state=RUNNABLE, 
group=TGRP-ChaosMonkeySafeLeaderTest]         at 
java.security.AccessController.doPrivileged(Native Method)         at 
com.carrotsearch.ant.tasks.junit4.events.Serializer.flushQueue(Serializer.java:114)
         at 
com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:99)
         at 
com.carrotsearch.ant.tasks.junit4.slave.SlaveMain$3$2.write(SlaveMain.java:459) 
        at 
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)         
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)         at 
java.io.PrintStream.flush(PrintStream.java:338)         at 
java.io.FilterOutputStream.flush(FilterOutputStream.java:140)         at 
java.io.PrintStream.write(PrintStream.java:482)         at 
sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)         at 
sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)         at 
sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)         at 
java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)         at 
java.io.PrintStream.write(PrintStream.java:527)         at 
java.io.PrintStream.print(PrintStream.java:669)         at 
java.io.PrintStream.println(PrintStream.java:823)         at 
java.lang.Throwable$WrappedPrintStream.println(Throwable.java:748)         at 
java.lang.Throwable.printStackTrace(Throwable.java:658)         at 
java.lang.Throwable.printStackTrace(Throwable.java:643)         at 
java.lang.Throwable.printStackTrace(Throwable.java:634)         at 
org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:121)
    3) Thread[id=48174, 
name=StoppableIndexingThread-SendThread(127.0.0.1:55407), state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderTest]         at java.lang.Thread.sleep(Native 
Method)         at 
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)    4) 
Thread[id=48156, name=StoppableIndexingThread, state=RUNNABLE, 
group=TGRP-ChaosMonkeySafeLeaderTest]         at 
org.apache.solr.common.util.JavaBinCodec.writeExternString(JavaBinCodec.java:1072)
         at 
org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:615)
         at 
org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)  
       at 
org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)        
 at 
org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)   
      at 
org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)  
       at 
org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)        
 at 
org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)   
      at 
org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)  
       at 
org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)        
 at 
org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)  
       at 
org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)  
       at 
org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)        
 at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)     
    at 
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)
         at 
org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)
         at 
org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)
         at 
org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)
         at 
org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:362)
         at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:231)
         at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:222)
         at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)         
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)       
  at 
org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)
         at 
org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)
    5) Thread[id=48176, name=zkCallback-1500-thread-1, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)    6) Thread[id=48256, 
name=zkCallback-1500-thread-4, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)    7) Thread[id=48254, 
name=zkCallback-1500-thread-2, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie 
threads that couldn't be terminated:
   1) Thread[id=48255, name=zkCallback-1500-thread-3, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=48157, name=StoppableIndexingThread, state=RUNNABLE, 
group=TGRP-ChaosMonkeySafeLeaderTest]
        at java.security.AccessController.doPrivileged(Native Method)
        at 
com.carrotsearch.ant.tasks.junit4.events.Serializer.flushQueue(Serializer.java:114)
        at 
com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:99)
        at 
com.carrotsearch.ant.tasks.junit4.slave.SlaveMain$3$2.write(SlaveMain.java:459)
        at 
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
        at java.io.PrintStream.flush(PrintStream.java:338)
        at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)
        at java.io.PrintStream.write(PrintStream.java:482)
        at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
        at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
        at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)
        at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)
        at java.io.PrintStream.write(PrintStream.java:527)
        at java.io.PrintStream.print(PrintStream.java:669)
        at java.io.PrintStream.println(PrintStream.java:823)
        at java.lang.Throwable$WrappedPrintStream.println(Throwable.java:748)
        at java.lang.Throwable.printStackTrace(Throwable.java:658)
        at java.lang.Throwable.printStackTrace(Throwable.java:643)
        at java.lang.Throwable.printStackTrace(Throwable.java:634)
        at 
org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:121)
   3) Thread[id=48174, 
name=StoppableIndexingThread-SendThread(127.0.0.1:55407), state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)
   4) Thread[id=48156, name=StoppableIndexingThread, state=RUNNABLE, 
group=TGRP-ChaosMonkeySafeLeaderTest]
        at 
org.apache.solr.common.util.JavaBinCodec.writeExternString(JavaBinCodec.java:1072)
        at 
org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:615)
        at 
org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)
        at 
org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at 
org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)
        at 
org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)
        at 
org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at 
org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)
        at 
org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)
        at 
org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at 
org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)
        at 
org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)
        at 
org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at 
org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)
        at 
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)
        at 
org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)
        at 
org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)
        at 
org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:362)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:231)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:222)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
        at 
org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)
        at 
org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)
   5) Thread[id=48176, name=zkCallback-1500-thread-1, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=48256, name=zkCallback-1500-thread-4, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   7) Thread[id=48254, name=zkCallback-1500-thread-2, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([304B34BAA0DA99AA]:0)


FAILED:  org.apache.solr.ltr.TestLTROnSolrCloud.testSimpleQuery

Error Message:
Path not found: /responseHeader/status

Stack Trace:
java.lang.RuntimeException: Path not found: /responseHeader/status
        at 
__randomizedtesting.SeedInfo.seed([9ADFB093F925742D:EFD2708FE93FB2F8]:0)
        at org.apache.solr.util.RestTestBase.assertJPut(RestTestBase.java:329)
        at org.apache.solr.util.RestTestBase.assertJPut(RestTestBase.java:271)
        at 
org.apache.solr.ltr.TestRerankBase.loadFeature(TestRerankBase.java:286)
        at 
org.apache.solr.ltr.TestLTROnSolrCloud.loadModelsAndFeatures(TestLTROnSolrCloud.java:184)
        at 
org.apache.solr.ltr.TestLTROnSolrCloud.setupSolrCluster(TestLTROnSolrCloud.java:138)
        at 
org.apache.solr.ltr.TestLTROnSolrCloud.setUp(TestLTROnSolrCloud.java:58)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:941)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test

Error Message:
The Monkey ran for over 45 seconds and no jetties were stopped - this is worth 
investigating!

Stack Trace:
java.lang.AssertionError: The Monkey ran for over 45 seconds and no jetties 
were stopped - this is worth investigating!
        at 
__randomizedtesting.SeedInfo.seed([304B34BAA0DA99AA:B81F0B600E26F452]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:587)
        at 
org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test(ChaosMonkeySafeLeaderTest.java:133)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.ClusterStateUpdateTest.testCoreRegistration

Error Message:
expected:<http[]://127.0.0.1:52825/s...> but 
was:<http[s]://127.0.0.1:52825/s...>

Stack Trace:
org.junit.ComparisonFailure: expected:<http[]://127.0.0.1:52825/s...> but 
was:<http[s]://127.0.0.1:52825/s...>
        at 
__randomizedtesting.SeedInfo.seed([304B34BAA0DA99AA:8EC05215D9A0979F]:0)
        at org.junit.Assert.assertEquals(Assert.java:125)
        at org.junit.Assert.assertEquals(Assert.java:147)
        at 
org.apache.solr.cloud.ClusterStateUpdateTest.testCoreRegistration(ClusterStateUpdateTest.java:104)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.ShardSplitTest.testSplitWithChaosMonkey

Error Message:
shard2 is not consistent.  Got 26 from 
http://127.0.0.1:45177/pmc/az/collection1_shard2_replica_t0 (previous client) 
and got 40 from http://127.0.0.1:42707/pmc/az/collection1_shard2_replica_t1

Stack Trace:
java.lang.AssertionError: shard2 is not consistent.  Got 26 from 
http://127.0.0.1:45177/pmc/az/collection1_shard2_replica_t0 (previous client) 
and got 40 from http://127.0.0.1:42707/pmc/az/collection1_shard2_replica_t1
        at 
__randomizedtesting.SeedInfo.seed([304B34BAA0DA99AA:BB6CE76BE1DC322E]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1319)
        at 
org.apache.solr.cloud.ShardSplitTest.testSplitWithChaosMonkey(ShardSplitTest.java:474)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.hdfs.HdfsUnloadDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=5701, 
name=testExecutor-1956-thread-7, state=RUNNABLE, 
group=TGRP-HdfsUnloadDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=5701, name=testExecutor-1956-thread-7, 
state=RUNNABLE, group=TGRP-HdfsUnloadDistributedZkTest]
Caused by: 
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at http://127.0.0.1:36170: Cannot create 1 new replicas for 
collection multiunload given the current number of live nodes and a 
maxShardsPerNode of 1
        at __randomizedtesting.SeedInfo.seed([304B34BAA0DA99AA]:0)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:233)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:222)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
        at 
org.apache.solr.cloud.BasicDistributedZkTest.lambda$createCores$0(BasicDistributedZkTest.java:598)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 11626 lines...]
   [junit4] Suite: org.apache.solr.cloud.ShardSplitTest
   [junit4]   2> Creating dataDir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_304B34BAA0DA99AA-001/init-core-data-001
   [junit4]   2> 222127 WARN  
(SUITE-ShardSplitTest-seed#[304B34BAA0DA99AA]-worker) [    ] 
o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
   [junit4]   2> 222127 INFO  
(SUITE-ShardSplitTest-seed#[304B34BAA0DA99AA]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) 
w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 222129 INFO  
(SUITE-ShardSplitTest-seed#[304B34BAA0DA99AA]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: 
@org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 222129 INFO  
(SUITE-ShardSplitTest-seed#[304B34BAA0DA99AA]-worker) [    ] 
o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /pmc/az
   [junit4]   2> 222133 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 222133 INFO  (Thread-1383) [    ] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 222133 INFO  (Thread-1383) [    ] o.a.s.c.ZkTestServer 
Starting server
   [junit4]   2> 222136 ERROR (Thread-1383) [    ] o.a.z.s.ZooKeeperServer 
ZKShutdownHandler is not registered, so ZooKeeper server won't take any action 
on ERROR or SHUTDOWN server state changes
   [junit4]   2> 222233 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.c.ZkTestServer start zk server on port:57279
   [junit4]   2> 222256 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
 to /configs/conf1/solrconfig.xml
   [junit4]   2> 222262 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml
 to /configs/conf1/schema.xml
   [junit4]   2> 222264 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
 to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 222266 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
 to /configs/conf1/stopwords.txt
   [junit4]   2> 222267 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt
 to /configs/conf1/protwords.txt
   [junit4]   2> 222269 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml
 to /configs/conf1/currency.xml
   [junit4]   2> 222271 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
 to /configs/conf1/enumsConfig.xml
   [junit4]   2> 222272 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
 to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 222274 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
 to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 222276 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
 to /configs/conf1/old_synonyms.txt
   [junit4]   2> 222277 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
 to /configs/conf1/synonyms.txt
   [junit4]   2> 222279 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless 
explicitly asked otherwise
   [junit4]   2> 222393 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 222394 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@1e6eaf9c{/pmc/az,null,AVAILABLE}
   [junit4]   2> 222401 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.e.j.s.AbstractConnector Started 
ServerConnector@683d0fba{HTTP/1.1,[http/1.1]}{127.0.0.1:50070}
   [junit4]   2> 222401 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.e.j.s.Server Started @225252ms
   [junit4]   2> 222401 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_304B34BAA0DA99AA-001/tempDir-001/control/data,
 hostContext=/pmc/az, hostPort=50070, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_304B34BAA0DA99AA-001/control-001/cores}
   [junit4]   2> 222401 ERROR 
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may 
be missing or incomplete.
   [junit4]   2> 222401 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ 
version 7.0.0
   [junit4]   2> 222402 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on 
port null
   [junit4]   2> 222402 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 222402 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2017-06-23T10:25:25.102Z
   [junit4]   2> 222406 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 222406 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_304B34BAA0DA99AA-001/control-001/solr.xml
   [junit4]   2> 222412 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.c.SolrXmlConfig MBean server found: 
com.sun.jmx.mbeanserver.JmxMBeanServer@71aaca9e, but no JMX reporters were 
configured - adding default JMX reporter.
   [junit4]   2> 222415 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:57279/solr
   [junit4]   2> 222443 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) 
[n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 222444 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) 
[n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.OverseerElectionContext I am going to 
be the leader 127.0.0.1:50070_pmc%2Faz
   [junit4]   2> 222445 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) 
[n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.Overseer Overseer 
(id=98186921563521028-127.0.0.1:50070_pmc%2Faz-n_0000000000) starting
   [junit4]   2> 222454 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) 
[n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:50070_pmc%2Faz
   [junit4]   2> 222457 INFO  
(zkCallback-229-thread-1-processing-n:127.0.0.1:50070_pmc%2Faz) 
[n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.c.ZkStateReader Updated live nodes 
from ZooKeeper... (0) -> (1)
   [junit4]   2> 222531 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) 
[n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@71aaca9e
   [junit4]   2> 222537 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) 
[n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@71aaca9e
   [junit4]   2> 222537 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) 
[n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@71aaca9e
   [junit4]   2> 222538 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) 
[n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_304B34BAA0DA99AA-001/control-001/cores
   [junit4]   2> 222559 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 222560 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:57279/solr 
ready
   [junit4]   2> 222563 INFO  (qtp1503329364-2843) [n:127.0.0.1:50070_pmc%2Faz  
  ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params 
pullReplicas=0&replicationFactor=1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&tlogReplicas=0&createNodeSet=127.0.0.1:50070_pmc%252Faz&wt=javabin&version=2
 and sendToOCPQueue=true
   [junit4]   2> 222567 INFO  
(OverseerThreadFactory-989-thread-1-processing-n:127.0.0.1:50070_pmc%2Faz) 
[n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.CreateCollectionCmd Create collection 
control_collection
   [junit4]   2> 222568 INFO  
(OverseerThreadFactory-989-thread-1-processing-n:127.0.0.1:50070_pmc%2Faz) 
[n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.CreateCollectionCmd Only one config 
set found in zk - using it:conf1
   [junit4]   2> 222673 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz  
  ] o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 222673 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz  
  ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 
transient cores
   [junit4]   2> 222778 INFO  
(zkCallback-229-thread-1-processing-n:127.0.0.1:50070_pmc%2Faz) 
[n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.c.ZkStateReader A cluster state 
change: [WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/control_collection/state.json] for collection 
[control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 223691 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 223706 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 223803 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 223834 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' 
using configuration from collection control_collection, trusted=true
   [junit4]   2> 223835 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.core.control_collection.shard1.replica_n1' (registry 
'solr.core.control_collection.shard1.replica_n1') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@71aaca9e
   [junit4]   2> 223835 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 223835 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore 
at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_304B34BAA0DA99AA-001/control-001/cores/control_collection_shard1_replica_n1],
 
dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_304B34BAA0DA99AA-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 223840 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: 
minMergeSize=0, mergeFactor=10, maxMergeSize=679371625, 
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, 
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=0.1]
   [junit4]   2> 223872 WARN  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = 
requestHandler,name = /dump,class = DumpRequestHandler,attributes = 
{initParams=a, name=/dump, class=DumpRequestHandler},args = 
{defaults={a=A,b=B}}}
   [junit4]   2> 223965 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.u.UpdateHandler Using UpdateLog implementation: 
org.apache.solr.update.UpdateLog
   [junit4]   2> 223965 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 223966 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 223966 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 223970 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: 
minMergeSize=0, mergeFactor=10, maxMergeSize=1738676577, 
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, 
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=0.1]
   [junit4]   2> 223971 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@2fe2544c[control_collection_shard1_replica_n1] main]
   [junit4]   2> 223972 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf1
   [junit4]   2> 223972 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 223974 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 223974 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.u.UpdateLog Could not find max version in index or recent updates, using 
new clock 1570990746945716224
   [junit4]   2> 223977 INFO  
(searcherExecutor-992-thread-1-processing-n:127.0.0.1:50070_pmc%2Faz 
x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) 
[n:127.0.0.1:50070_pmc%2Faz c:control_collection s:shard1  
x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore 
[control_collection_shard1_replica_n1] Registered new searcher 
Searcher@2fe2544c[control_collection_shard1_replica_n1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 223982 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 223982 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 223982 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.SyncStrategy Sync replicas to 
http://127.0.0.1:50070/pmc/az/control_collection_shard1_replica_n1/
   [junit4]   2> 223982 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 223982 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.SyncStrategy 
http://127.0.0.1:50070/pmc/az/control_collection_shard1_replica_n1/ has no 
replicas
   [junit4]   2> 223982 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.ShardLeaderElectionContext Found all replicas participating in 
election, clear LIR
   [junit4]   2> 223989 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.ShardLeaderElectionContext I am the new leader: 
http://127.0.0.1:50070/pmc/az/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 224092 INFO  
(zkCallback-229-thread-1-processing-n:127.0.0.1:50070_pmc%2Faz) 
[n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.c.ZkStateReader A cluster state 
change: [WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/control_collection/state.json] for collection 
[control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 224140 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 224143 INFO  (qtp1503329364-2845) [n:127.0.0.1:50070_pmc%2Faz 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT}
 status=0 QTime=1470
   [junit4]   2> 224148 INFO  (qtp1503329364-2843) [n:127.0.0.1:50070_pmc%2Faz  
  ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at 
most 30 seconds. Check all shard replicas
   [junit4]   2> 224246 INFO  
(zkCallback-229-thread-1-processing-n:127.0.0.1:50070_pmc%2Faz) 
[n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.c.ZkStateReader A cluster state 
change: [WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/control_collection/state.json] for collection 
[control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 224569 INFO  
(OverseerCollectionConfigSetProcessor-98186921563521028-127.0.0.1:50070_pmc%2Faz-n_0000000000)
 [n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.OverseerTaskQueue Response ZK path: 
/overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may 
have disconnected from ZooKeeper
   [junit4]   2> 225148 INFO  (qtp1503329364-2843) [n:127.0.0.1:50070_pmc%2Faz  
  ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={pullReplicas=0&replicationFactor=1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&tlogReplicas=0&createNodeSet=127.0.0.1:50070_pmc%252Faz&wt=javabin&version=2}
 status=0 QTime=2585
   [junit4]   2> 225155 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 225156 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:57279/solr 
ready
   [junit4]   2> 225156 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection 
loss:false
   [junit4]   2> 225158 INFO  (qtp1503329364-2840) [n:127.0.0.1:50070_pmc%2Faz  
  ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params 
pullReplicas=0&replicationFactor=1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&tlogReplicas=0&createNodeSet=&stateFormat=1&wt=javabin&version=2
 and sendToOCPQueue=true
   [junit4]   2> 225160 INFO  
(OverseerThreadFactory-989-thread-2-processing-n:127.0.0.1:50070_pmc%2Faz) 
[n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.CreateCollectionCmd Create collection 
collection1
   [junit4]   2> 225161 INFO  
(OverseerThreadFactory-989-thread-2-processing-n:127.0.0.1:50070_pmc%2Faz) 
[n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.CreateCollectionCmd Only one config 
set found in zk - using it:conf1
   [junit4]   2> 225161 WARN  
(OverseerThreadFactory-989-thread-2-processing-n:127.0.0.1:50070_pmc%2Faz) 
[n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.CreateCollectionCmd It is unusual to 
create a collection (collection1) without cores.
   [junit4]   2> 225366 INFO  (qtp1503329364-2840) [n:127.0.0.1:50070_pmc%2Faz  
  ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at 
most 30 seconds. Check all shard replicas
   [junit4]   2> 225366 INFO  (qtp1503329364-2840) [n:127.0.0.1:50070_pmc%2Faz  
  ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={pullReplicas=0&replicationFactor=1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&tlogReplicas=0&createNodeSet=&stateFormat=1&wt=javabin&version=2}
 status=0 QTime=208
   [junit4]   2> 225485 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_304B34BAA0DA99AA-001/shard-1-001
 of type NRT
   [junit4]   2> 225485 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 225487 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@70d71f4c{/pmc/az,null,AVAILABLE}
   [junit4]   2> 225487 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.e.j.s.AbstractConnector Started 
ServerConnector@29a1bb8a{HTTP/1.1,[http/1.1]}{127.0.0.1:48501}
   [junit4]   2> 225488 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.e.j.s.Server Started @228339ms
   [junit4]   2> 225488 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_304B34BAA0DA99AA-001/tempDir-001/jetty1,
 solrconfig=solrconfig.xml, hostContext=/pmc/az, hostPort=48501, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_304B34BAA0DA99AA-001/shard-1-001/cores}
   [junit4]   2> 225488 ERROR 
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may 
be missing or incomplete.
   [junit4]   2> 225488 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ 
version 7.0.0
   [junit4]   2> 225488 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on 
port null
   [junit4]   2> 225488 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 225489 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2017-06-23T10:25:28.189Z
   [junit4]   2> 225493 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 225493 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_304B34BAA0DA99AA-001/shard-1-001/solr.xml
   [junit4]   2> 225499 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.c.SolrXmlConfig MBean server found: 
com.sun.jmx.mbeanserver.JmxMBeanServer@71aaca9e, but no JMX reporters were 
configured - adding default JMX reporter.
   [junit4]   2> 225502 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:57279/solr
   [junit4]   2> 225514 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) 
[n:127.0.0.1:48501_pmc%2Faz    ] o.a.s.c.c.ZkStateReader Updated live nodes 
from ZooKeeper... (0) -> (1)
   [junit4]   2> 225517 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) 
[n:127.0.0.1:48501_pmc%2Faz    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 225520 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) 
[n:127.0.0.1:48501_pmc%2Faz    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:48501_pmc%2Faz
   [junit4]   2> 225521 INFO  
(zkCallback-229-thread-1-processing-n:127.0.0.1:50070_pmc%2Faz) 
[n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.c.ZkStateReader Updated live nodes 
from ZooKeeper... (1) -> (2)
   [junit4]   2> 225523 INFO  (zkCallback-236-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 225524 INFO  
(zkCallback-241-thread-1-processing-n:127.0.0.1:48501_pmc%2Faz) 
[n:127.0.0.1:48501_pmc%2Faz    ] o.a.s.c.c.ZkStateReader Updated live nodes 
from ZooKeeper... (1) -> (2)
   [junit4]   2> 225672 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) 
[n:127.0.0.1:48501_pmc%2Faz    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@71aaca9e
   [junit4]   2> 225678 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) 
[n:127.0.0.1:48501_pmc%2Faz    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@71aaca9e
   [junit4]   2> 225678 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) 
[n:127.0.0.1:48501_pmc%2Faz    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@71aaca9e
   [junit4]   2> 225680 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) 
[n:127.0.0.1:48501_pmc%2Faz    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_304B34BAA0DA99AA-001/shard-1-001/cores
   [junit4]   2> 225706 INFO  (qtp1503329364-2841) [n:127.0.0.1:50070_pmc%2Faz  
  ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with 
params 
node=127.0.0.1:48501_pmc%252Faz&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2
 and sendToOCPQueue=true
   [junit4]   2> 225708 INFO  
(OverseerThreadFactory-989-thread-3-processing-n:127.0.0.1:50070_pmc%2Faz) 
[n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.AddReplicaCmd addReplica() : {
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "node":"127.0.0.1:48501_pmc%2Faz",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "operation":"addreplica"}
   [junit4]   2> 225708 INFO  
(OverseerThreadFactory-989-thread-3-processing-n:127.0.0.1:50070_pmc%2Faz) 
[n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.AddReplicaCmd Node Identified 
127.0.0.1:48501_pmc%2Faz for creating new replica
   [junit4]   2> 225709 INFO  
(OverseerCollectionConfigSetProcessor-98186921563521028-127.0.0.1:50070_pmc%2Faz-n_0000000000)
 [n:127.0.0.1:50070_pmc%2Faz    ] o.a.s.c.OverseerTaskQueue Response ZK path: 
/overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may 
have disconnected from ZooKeeper
   [junit4]   2> 225711 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz  
  ] o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n0&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 225711 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz  
  ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 
transient cores
   [junit4]   2> 226728 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz 
c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 226742 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz 
c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.IndexSchema 
[collection1_shard2_replica_n0] Schema name=test
   [junit4]   2> 226829 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz 
c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.IndexSchema 
Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 226849 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz 
c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.CoreContainer 
Creating SolrCore 'collection1_shard2_replica_n0' using configuration from 
collection collection1, trusted=true
   [junit4]   2> 226849 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz 
c:collection1 s:shard2  x:collection1_shard2_replica_n0] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.core.collection1.shard2.replica_n0' (registry 
'solr.core.collection1.shard2.replica_n0') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@71aaca9e
   [junit4]   2> 226849 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz 
c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SolrCore 
solr.RecoveryStrategy.Builder
   [junit4]   2> 226850 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz 
c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SolrCore 
[[collection1_shard2_replica_n0] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_304B34BAA0DA99AA-001/shard-1-001/cores/collection1_shard2_replica_n0],
 
dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_304B34BAA0DA99AA-001/shard-1-001/cores/collection1_shard2_replica_n0/data/]
   [junit4]   2> 226854 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz 
c:collection1 s:shard2  x:collection1_shard2_replica_n0] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: 
minMergeSize=0, mergeFactor=10, maxMergeSize=679371625, 
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, 
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=0.1]
   [junit4]   2> 226866 WARN  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz 
c:collection1 s:shard2  x:collection1_shard2_replica_n0] 
o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = 
requestHandler,name = /dump,class = DumpRequestHandler,attributes = 
{initParams=a, name=/dump, class=DumpRequestHandler},args = 
{defaults={a=A,b=B}}}
   [junit4]   2> 226920 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz 
c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.UpdateHandler 
Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 226920 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz 
c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.UpdateLog 
Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 
maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 226921 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz 
c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.CommitTracker 
Hard AutoCommit: disabled
   [junit4]   2> 226921 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz 
c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.CommitTracker 
Soft AutoCommit: disabled
   [junit4]   2> 226923 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz 
c:collection1 s:shard2  x:collection1_shard2_replica_n0] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: 
minMergeSize=0, mergeFactor=10, maxMergeSize=1738676577, 
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, 
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=0.1]
   [junit4]   2> 226924 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz 
c:collection1 s:shard2  x:collection1_shard2_replica_n0] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@36f99ff0[collection1_shard2_replica_n0] main]
   [junit4]   2> 226926 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz 
c:collection1 s:shard2  x:collection1_shard2_replica_n0] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf1
   [junit4]   2> 226926 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz 
c:collection1 s:shard2  x:collection1_shard2_replica_n0] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 226927 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz 
c:collection1 s:shard2  x:collection1_shard2_replica_n0] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 226929 INFO  
(searcherExecutor-1003-thread-1-processing-n:127.0.0.1:48501_pmc%2Faz 
x:collection1_shard2_replica_n0 s:shard2 c:collection1) 
[n:127.0.0.1:48501_pmc%2Faz c:collection1 s:shard2  
x:collection1_shard2_replica_n0] o.a.s.c.SolrCore 
[collection1_shard2_replica_n0] Registered new searcher 
Searcher@36f99ff0[collection1_shard2_replica_n0] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 226929 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz 
c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.UpdateLog 
Could not find max version in index or recent updates, using new clock 
1570990750044258304
   [junit4]   2> 226936 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz 
c:collection1 s:shard2  x:collection1_shard2_replica_n0] 
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 226936 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz 
c:collection1 s:shard2  x:collection1_shard2_replica_n0] 
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 226936 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz 
c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SyncStrategy 
Sync replicas to http://127.0.0.1:48501/pmc/az/collection1_shard2_replica_n0/
   [junit4]   2> 226936 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz 
c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SyncStrategy 
Sync Success - now sync replicas to me
   [junit4]   2> 226936 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz 
c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SyncStrategy 
http://127.0.0.1:48501/pmc/az/collection1_shard2_replica_n0/ has no replicas
   [junit4]   2> 226936 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz 
c:collection1 s:shard2  x:collection1_shard2_replica_n0] 
o.a.s.c.ShardLeaderElectionContext Found all replicas participating in 
election, clear LIR
   [junit4]   2> 226941 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz 
c:collection1 s:shard2  x:collection1_shard2_replica_n0] 
o.a.s.c.ShardLeaderElectionContext I am the new leader: 
http://127.0.0.1:48501/pmc/az/collection1_shard2_replica_n0/ shard2
   [junit4]   2> 227092 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz 
c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ZkController I 
am the leader, no recovery necessary
   [junit4]   2> 227095 INFO  (qtp1038909219-2884) [n:127.0.0.1:48501_pmc%2Faz 
c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.HttpSolrCall 
[admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n0&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT}
 status=0 QTime=1384
   [junit4]   2> 227098 INFO  (qtp1503329364-2841) [n:127.0.0.1:50070_pmc%2Faz  
  ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={node=127.0.0.1:48501_pmc%252Faz&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2}
 status=0 QTime=1392
   [junit4]   2> 227216 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_304B34BAA0DA99AA-001/shard-2-001
 of type NRT
   [junit4]   2> 227217 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 227218 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@1696418d{/pmc/az,null,AVAILABLE}
   [junit4]   2> 227219 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.e.j.s.AbstractConnector Started 
ServerConnector@6b1eb9aa{HTTP/1.1,[http/1.1]}{127.0.0.1:42883}
   [junit4]   2> 227219 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.e.j.s.Server Started @230070ms
   [junit4]   2> 227219 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_304B34BAA0DA99AA-001/tempDir-001/jetty2,
 solrconfig=solrconfig.xml, hostContext=/pmc/az, hostPort=42883, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_304B34BAA0DA99AA-001/shard-2-001/cores}
   [junit4]   2> 227222 ERROR 
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may 
be missing or incomplete.
   [junit4]   2> 227223 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ 
version 7.0.0
   [junit4]   2> 227223 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on 
port null
   [junit4]   2> 227223 INFO  
(TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[304B34BAA0DA99AA]) [ 
   ] o.a.s.s.SolrDispatchFilter \

[...truncated too long message...]

n1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 28581 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from 
client sessionid 0x15cd4ca75cd0024, likely client has closed socket
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 28582 INFO  
(zkCallback-78-thread-2-processing-n:127.0.0.1:52790_solr) 
[n:127.0.0.1:52790_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (3) -> (2)
   [junit4]   2> 28582 INFO  
(zkCallback-75-thread-1-processing-n:127.0.0.1:52533_solr) 
[n:127.0.0.1:52533_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (3) -> (2)
   [junit4]   2> 28594 INFO  (coreCloseExecutor-225-thread-1) 
[n:127.0.0.1:52533_solr c:collection1 s:shard1 r:core_node1 
x:collection1_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric 
reporters for registry=solr.collection.collection1.shard1.leader, tag=248948481
   [junit4]   2> 28595 INFO  
(zkCallback-75-thread-2-processing-n:127.0.0.1:52533_solr) 
[n:127.0.0.1:52533_solr    ] o.a.s.c.OverseerElectionContext I am going to be 
the leader 127.0.0.1:52533_solr
   [junit4]   2> 28595 INFO  
(zkCallback-75-thread-1-processing-n:127.0.0.1:52533_solr) 
[n:127.0.0.1:52533_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (2) -> (1)
   [junit4]   2> 29928 WARN  
(zkCallback-73-thread-2-processing-n:127.0.0.1:48414_solr) 
[n:127.0.0.1:48414_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, 
but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 29928 INFO  (jetty-closer-20-thread-11) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@2fb48412{/solr,null,UNAVAILABLE}
   [junit4]   2> 29930 WARN  
(zkCallback-76-thread-2-processing-n:127.0.0.1:34517_solr) 
[n:127.0.0.1:34517_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, 
but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 29931 WARN  
(zkCallback-68-thread-2-processing-n:127.0.0.1:60829_solr) 
[n:127.0.0.1:60829_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, 
but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 29931 INFO  (jetty-closer-20-thread-8) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@342d4886{/solr,null,UNAVAILABLE}
   [junit4]   2> 29931 INFO  (jetty-closer-20-thread-2) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@5abe9b4f{/solr,null,UNAVAILABLE}
   [junit4]   2> 29931 WARN  
(zkCallback-71-thread-2-processing-n:127.0.0.1:36159_solr) 
[n:127.0.0.1:36159_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, 
but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 29932 INFO  (jetty-closer-20-thread-3) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@1808bbbc{/solr,null,UNAVAILABLE}
   [junit4]   2> 29990 WARN  
(zkCallback-69-thread-1-processing-n:127.0.0.1:60858_solr) 
[n:127.0.0.1:60858_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, 
but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 29990 WARN  
(zkCallback-80-thread-2-processing-n:127.0.0.1:39013_solr) 
[n:127.0.0.1:39013_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, 
but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 29991 INFO  (jetty-closer-20-thread-9) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@23191a50{/solr,null,UNAVAILABLE}
   [junit4]   2> 29991 INFO  (jetty-closer-20-thread-7) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@7d2232aa{/solr,null,UNAVAILABLE}
   [junit4]   2> 30080 WARN  
(zkCallback-81-thread-2-processing-n:127.0.0.1:32772_solr) 
[n:127.0.0.1:32772_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, 
but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for 
/collections/collection1/state.json]
   [junit4]   2> 30080 WARN  
(zkCallback-81-thread-1-processing-n:127.0.0.1:32772_solr) 
[n:127.0.0.1:32772_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, 
but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for 
/collections/collection1/state.json]
   [junit4]   2> 30080 INFO  (jetty-closer-20-thread-1) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@2d605775{/solr,null,UNAVAILABLE}
   [junit4]   2> 30093 WARN  
(zkCallback-78-thread-2-processing-n:127.0.0.1:52790_solr) 
[n:127.0.0.1:52790_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, 
but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 30094 INFO  (jetty-closer-20-thread-5) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@3f1125f5{/solr,null,UNAVAILABLE}
   [junit4]   2> 30097 WARN  
(zkCallback-75-thread-1-processing-n:127.0.0.1:52533_solr) 
[n:127.0.0.1:52533_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, 
but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 30098 INFO  (jetty-closer-20-thread-10) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@1bb7cdbe{/solr,null,UNAVAILABLE}
   [junit4]   2> 30099 ERROR 
(TEST-TestLTROnSolrCloud.testSimpleQuery-seed#[9ADFB093F925742D]) [    ] 
o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper 
server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 30101 INFO  
(TEST-TestLTROnSolrCloud.testSimpleQuery-seed#[9ADFB093F925742D]) [    ] 
o.a.s.c.ZkTestServer connecting to 127.0.0.1:45671 45671
   [junit4]   2> 30189 INFO  (Thread-46) [    ] o.a.s.c.ZkTestServer connecting 
to 127.0.0.1:45671 45671
   [junit4]   2> 30190 WARN  (Thread-46) [    ] o.a.s.c.ZkTestServer Watch 
limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2>        14      /solr/aliases.json
   [junit4]   2>        14      /solr/clusterprops.json
   [junit4]   2>        13      /solr/security.json
   [junit4]   2>        3       /solr/configs/conf1
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2>        16      /solr/collections/collection1/state.json
   [junit4]   2>        14      /solr/clusterstate.json
   [junit4]   2>        3       
/solr/overseer_elect/election/98187257920159777-127.0.0.1:52790_solr-n_0000000000
   [junit4]   2>        2       
/solr/overseer_elect/election/98187257920159779-127.0.0.1:39013_solr-n_0000000004
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2>        14      /solr/live_nodes
   [junit4]   2>        14      /solr/collections
   [junit4]   2> 
   [junit4]   2> 30191 INFO  
(TEST-TestLTROnSolrCloud.testSimpleQuery-seed#[9ADFB093F925742D]) [    ] 
o.a.s.SolrTestCaseJ4 ###Ending testSimpleQuery
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 
'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestLTROnSolrCloud 
-Dtests.method=testSimpleQuery -Dtests.seed=9ADFB093F925742D 
-Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt
 -Dtests.locale=es-PA -Dtests.timezone=Pacific/Kwajalein -Dtests.asserts=true 
-Dtests.file.encoding=US-ASCII
   [junit4] ERROR   10.3s J0 | TestLTROnSolrCloud.testSimpleQuery <<<
   [junit4]    > Throwable #1: java.lang.RuntimeException: Path not found: 
/responseHeader/status
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([9ADFB093F925742D:EFD2708FE93FB2F8]:0)
   [junit4]    >        at 
org.apache.solr.util.RestTestBase.assertJPut(RestTestBase.java:329)
   [junit4]    >        at 
org.apache.solr.util.RestTestBase.assertJPut(RestTestBase.java:271)
   [junit4]    >        at 
org.apache.solr.ltr.TestRerankBase.loadFeature(TestRerankBase.java:286)
   [junit4]    >        at 
org.apache.solr.ltr.TestLTROnSolrCloud.loadModelsAndFeatures(TestLTROnSolrCloud.java:184)
   [junit4]    >        at 
org.apache.solr.ltr.TestLTROnSolrCloud.setupSolrCluster(TestLTROnSolrCloud.java:138)
   [junit4]    >        at 
org.apache.solr.ltr.TestLTROnSolrCloud.setUp(TestLTROnSolrCloud.java:58)
   [junit4]    >        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/contrib/solr-ltr/test/J0/temp/solr.ltr.TestLTROnSolrCloud_9ADFB093F925742D-001
   [junit4]   2> Jun 23, 2017 11:51:07 AM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked 
thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): 
{_version_=FST50, popularity=FSTOrd50, 
description=BlockTreeOrds(blocksize=128), id=FSTOrd50, 
text=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))),
 title=BlockTreeOrds(blocksize=128)}, docValues:{}, maxPointsInLeafNode=273, 
maxMBSortInHeap=7.989211293074257, sim=RandomSimilarity(queryNorm=true): {}, 
locale=es-PA, timezone=Pacific/Kwajalein
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 
1.8.0_131 (64-bit)/cpus=4,threads=1,free=165330216,total=332922880
   [junit4]   2> NOTE: All tests run in this JVM: [TestLTRReRankingPipeline, 
TestManagedFeatureStore, TestUserTermScorereQDF, TestFieldLengthFeature, 
TestLTRWithSort, TestSelectiveWeightCreation, TestLTROnSolrCloud]
   [junit4] Completed [26/33 (1!)] on J0 in 11.12s, 1 test, 1 error <<< 
FAILURES!

[...truncated 36821 lines...]
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Reply via email to