Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1335/

5 tests failed.
FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationHandlerTest

Error Message:
5 threads leaked from SUITE scope at 
org.apache.solr.cloud.CdcrReplicationHandlerTest:     1) Thread[id=4258, 
name=cdcr-update-log-synchronizer-1128-thread-1, state=WAITING, 
group=TGRP-CdcrReplicationHandlerTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)    2) Thread[id=4824, 
name=cdcr-update-log-synchronizer-1270-thread-1, state=WAITING, 
group=TGRP-CdcrReplicationHandlerTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)    3) Thread[id=4503, 
name=cdcr-update-log-synchronizer-1192-thread-1, state=WAITING, 
group=TGRP-CdcrReplicationHandlerTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)    4) Thread[id=5395, 
name=cdcr-update-log-synchronizer-1335-thread-1, state=WAITING, 
group=TGRP-CdcrReplicationHandlerTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)    5) Thread[id=5642, 
name=cdcr-update-log-synchronizer-1399-thread-1, state=WAITING, 
group=TGRP-CdcrReplicationHandlerTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from SUITE 
scope at org.apache.solr.cloud.CdcrReplicationHandlerTest: 
   1) Thread[id=4258, name=cdcr-update-log-synchronizer-1128-thread-1, 
state=WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=4824, name=cdcr-update-log-synchronizer-1270-thread-1, 
state=WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=4503, name=cdcr-update-log-synchronizer-1192-thread-1, 
state=WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=5395, name=cdcr-update-log-synchronizer-1335-thread-1, 
state=WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=5642, name=cdcr-update-log-synchronizer-1399-thread-1, 
state=WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([B0D1D400E98C113B]:0)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationHandlerTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) 
Thread[id=4258, name=cdcr-update-log-synchronizer-1128-thread-1, state=WAITING, 
group=TGRP-CdcrReplicationHandlerTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)    2) Thread[id=4824, 
name=cdcr-update-log-synchronizer-1270-thread-1, state=WAITING, 
group=TGRP-CdcrReplicationHandlerTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)    3) Thread[id=4503, 
name=cdcr-update-log-synchronizer-1192-thread-1, state=WAITING, 
group=TGRP-CdcrReplicationHandlerTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)    4) Thread[id=5395, 
name=cdcr-update-log-synchronizer-1335-thread-1, state=WAITING, 
group=TGRP-CdcrReplicationHandlerTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)    5) Thread[id=5642, 
name=cdcr-update-log-synchronizer-1399-thread-1, state=WAITING, 
group=TGRP-CdcrReplicationHandlerTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie 
threads that couldn't be terminated:
   1) Thread[id=4258, name=cdcr-update-log-synchronizer-1128-thread-1, 
state=WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=4824, name=cdcr-update-log-synchronizer-1270-thread-1, 
state=WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=4503, name=cdcr-update-log-synchronizer-1192-thread-1, 
state=WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=5395, name=cdcr-update-log-synchronizer-1335-thread-1, 
state=WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=5642, name=cdcr-update-log-synchronizer-1399-thread-1, 
state=WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([B0D1D400E98C113B]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrRequestHandlerTest

Error Message:
6 threads leaked from SUITE scope at 
org.apache.solr.cloud.CdcrRequestHandlerTest:     1) Thread[id=22741, 
name=cdcr-update-log-synchronizer-6330-thread-1, state=WAITING, 
group=TGRP-CdcrRequestHandlerTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)    2) Thread[id=23035, 
name=cdcr-update-log-synchronizer-6428-thread-1, state=WAITING, 
group=TGRP-CdcrRequestHandlerTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)    3) Thread[id=23023, 
name=cdcr-update-log-synchronizer-6422-thread-1, state=WAITING, 
group=TGRP-CdcrRequestHandlerTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)    4) Thread[id=23350, 
name=cdcr-update-log-synchronizer-6530-thread-1, state=WAITING, 
group=TGRP-CdcrRequestHandlerTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)    5) Thread[id=23362, 
name=cdcr-update-log-synchronizer-6536-thread-1, state=WAITING, 
group=TGRP-CdcrRequestHandlerTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)    6) Thread[id=22729, 
name=cdcr-update-log-synchronizer-6324-thread-1, state=WAITING, 
group=TGRP-CdcrRequestHandlerTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 6 threads leaked from SUITE 
scope at org.apache.solr.cloud.CdcrRequestHandlerTest: 
   1) Thread[id=22741, name=cdcr-update-log-synchronizer-6330-thread-1, 
state=WAITING, group=TGRP-CdcrRequestHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=23035, name=cdcr-update-log-synchronizer-6428-thread-1, 
state=WAITING, group=TGRP-CdcrRequestHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=23023, name=cdcr-update-log-synchronizer-6422-thread-1, 
state=WAITING, group=TGRP-CdcrRequestHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=23350, name=cdcr-update-log-synchronizer-6530-thread-1, 
state=WAITING, group=TGRP-CdcrRequestHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=23362, name=cdcr-update-log-synchronizer-6536-thread-1, 
state=WAITING, group=TGRP-CdcrRequestHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=22729, name=cdcr-update-log-synchronizer-6324-thread-1, 
state=WAITING, group=TGRP-CdcrRequestHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([B0D1D400E98C113B]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrRequestHandlerTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) 
Thread[id=22741, name=cdcr-update-log-synchronizer-6330-thread-1, 
state=WAITING, group=TGRP-CdcrRequestHandlerTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)    2) Thread[id=23035, 
name=cdcr-update-log-synchronizer-6428-thread-1, state=WAITING, 
group=TGRP-CdcrRequestHandlerTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)    3) Thread[id=23023, 
name=cdcr-update-log-synchronizer-6422-thread-1, state=WAITING, 
group=TGRP-CdcrRequestHandlerTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)    4) Thread[id=23350, 
name=cdcr-update-log-synchronizer-6530-thread-1, state=WAITING, 
group=TGRP-CdcrRequestHandlerTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)    5) Thread[id=23362, 
name=cdcr-update-log-synchronizer-6536-thread-1, state=WAITING, 
group=TGRP-CdcrRequestHandlerTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)    6) Thread[id=22729, 
name=cdcr-update-log-synchronizer-6324-thread-1, state=WAITING, 
group=TGRP-CdcrRequestHandlerTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie 
threads that couldn't be terminated:
   1) Thread[id=22741, name=cdcr-update-log-synchronizer-6330-thread-1, 
state=WAITING, group=TGRP-CdcrRequestHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=23035, name=cdcr-update-log-synchronizer-6428-thread-1, 
state=WAITING, group=TGRP-CdcrRequestHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=23023, name=cdcr-update-log-synchronizer-6422-thread-1, 
state=WAITING, group=TGRP-CdcrRequestHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=23350, name=cdcr-update-log-synchronizer-6530-thread-1, 
state=WAITING, group=TGRP-CdcrRequestHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=23362, name=cdcr-update-log-synchronizer-6536-thread-1, 
state=WAITING, group=TGRP-CdcrRequestHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=22729, name=cdcr-update-log-synchronizer-6324-thread-1, 
state=WAITING, group=TGRP-CdcrRequestHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([B0D1D400E98C113B]:0)


FAILED:  org.apache.solr.ltr.TestLTROnSolrCloud.testSimpleQuery

Error Message:
Path not found: /responseHeader/status

Stack Trace:
java.lang.RuntimeException: Path not found: /responseHeader/status
        at 
__randomizedtesting.SeedInfo.seed([C7CE671D17FFDB46:B2C3A70107E51D93]:0)
        at org.apache.solr.util.RestTestBase.assertJPut(RestTestBase.java:329)
        at org.apache.solr.util.RestTestBase.assertJPut(RestTestBase.java:271)
        at 
org.apache.solr.ltr.TestRerankBase.loadFeature(TestRerankBase.java:286)
        at 
org.apache.solr.ltr.TestLTROnSolrCloud.loadModelsAndFeatures(TestLTROnSolrCloud.java:184)
        at 
org.apache.solr.ltr.TestLTROnSolrCloud.setupSolrCluster(TestLTROnSolrCloud.java:138)
        at 
org.apache.solr.ltr.TestLTROnSolrCloud.setUp(TestLTROnSolrCloud.java:58)
        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:941)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 11721 lines...]
   [junit4] Suite: org.apache.solr.cloud.CdcrReplicationHandlerTest
   [junit4]   2> Creating dataDir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrReplicationHandlerTest_B0D1D400E98C113B-001/init-core-data-001
   [junit4]   2> 385259 WARN  
(SUITE-CdcrReplicationHandlerTest-seed#[B0D1D400E98C113B]-worker) [    ] 
o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 385259 INFO  
(SUITE-CdcrReplicationHandlerTest-seed#[B0D1D400E98C113B]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) 
w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 385261 INFO  
(SUITE-CdcrReplicationHandlerTest-seed#[B0D1D400E98C113B]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: 
@org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 385262 INFO  
(SUITE-CdcrReplicationHandlerTest-seed#[B0D1D400E98C113B]-worker) [    ] 
o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /io/hw
   [junit4]   2> 385267 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 385269 INFO  (Thread-1019) [    ] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 385269 INFO  (Thread-1019) [    ] o.a.s.c.ZkTestServer 
Starting server
   [junit4]   2> 385287 ERROR (Thread-1019) [    ] o.a.z.s.ZooKeeperServer 
ZKShutdownHandler is not registered, so ZooKeeper server won't take any action 
on ERROR or SHUTDOWN server state changes
   [junit4]   2> 385369 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.a.s.c.ZkTestServer start zk server on port:51040
   [junit4]   2> 385387 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-cdcr.xml
 to /configs/conf1/solrconfig.xml
   [junit4]   2> 385390 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml
 to /configs/conf1/schema.xml
   [junit4]   2> 385391 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
 to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 385393 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
 to /configs/conf1/stopwords.txt
   [junit4]   2> 385395 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt
 to /configs/conf1/protwords.txt
   [junit4]   2> 385396 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml
 to /configs/conf1/currency.xml
   [junit4]   2> 385397 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
 to /configs/conf1/enumsConfig.xml
   [junit4]   2> 385399 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
 to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 385401 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
 to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 385402 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
 to /configs/conf1/old_synonyms.txt
   [junit4]   2> 385404 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
 to /configs/conf1/synonyms.txt
   [junit4]   2> 385420 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from 
client sessionid 0x15cd8c1bed50002, likely client has closed socket
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 385420 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.a.s.SolrTestCaseJ4 ###Starting testFullReplication
   [junit4]   2> 385645 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 385647 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@344b5dec{/io/hw,null,AVAILABLE}
   [junit4]   2> 385652 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.e.j.s.AbstractConnector Started ServerConnector@3624c80e{SSL,[ssl, 
http/1.1]}{127.0.0.1:40256}
   [junit4]   2> 385652 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.e.j.s.Server Started @389523ms
   [junit4]   2> 385652 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {shards=shard1, 
hostContext=/io/hw, hostPort=40256, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrReplicationHandlerTest_B0D1D400E98C113B-001/jetty-001/cores}
   [junit4]   2> 385652 ERROR 
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may 
be missing or incomplete.
   [junit4]   2> 385653 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ 
version 7.0.0
   [junit4]   2> 385653 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on 
port null
   [junit4]   2> 385653 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 385653 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2017-06-24T06:19:55.338Z
   [junit4]   2> 385661 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 385661 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrReplicationHandlerTest_B0D1D400E98C113B-001/jetty-001/solr.xml
   [junit4]   2> 385671 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.a.s.c.SolrXmlConfig MBean server found: 
com.sun.jmx.mbeanserver.JmxMBeanServer@14bf4aab, but no JMX reporters were 
configured - adding default JMX reporter.
   [junit4]   2> 385675 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51040/solr
   [junit4]   2> 385721 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) 
[n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 385725 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) 
[n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.OverseerElectionContext I am going to 
be the leader 127.0.0.1:40256_io%2Fhw
   [junit4]   2> 385727 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) 
[n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.Overseer Overseer 
(id=98191618537226245-127.0.0.1:40256_io%2Fhw-n_0000000000) starting
   [junit4]   2> 385738 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) 
[n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:40256_io%2Fhw
   [junit4]   2> 385761 INFO  
(zkCallback-1215-thread-1-processing-n:127.0.0.1:40256_io%2Fhw) 
[n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 385823 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) 
[n:127.0.0.1:40256_io%2Fhw    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@14bf4aab
   [junit4]   2> 385829 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) 
[n:127.0.0.1:40256_io%2Fhw    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@14bf4aab
   [junit4]   2> 385829 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) 
[n:127.0.0.1:40256_io%2Fhw    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@14bf4aab
   [junit4]   2> 385830 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) 
[n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrReplicationHandlerTest_B0D1D400E98C113B-001/jetty-001/cores
   [junit4]   2> 385976 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 385988 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@4b68787d{/io/hw,null,AVAILABLE}
   [junit4]   2> 385990 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.e.j.s.AbstractConnector Started ServerConnector@5129940d{SSL,[ssl, 
http/1.1]}{127.0.0.1:39054}
   [junit4]   2> 385990 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.e.j.s.Server Started @389860ms
   [junit4]   2> 385990 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {shards=shard2, 
hostContext=/io/hw, hostPort=39054, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrReplicationHandlerTest_B0D1D400E98C113B-001/jetty-002/cores}
   [junit4]   2> 385990 ERROR 
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may 
be missing or incomplete.
   [junit4]   2> 385990 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ 
version 7.0.0
   [junit4]   2> 385990 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on 
port null
   [junit4]   2> 385990 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 385991 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2017-06-24T06:19:55.675Z
   [junit4]   2> 385994 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 385994 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrReplicationHandlerTest_B0D1D400E98C113B-001/jetty-002/solr.xml
   [junit4]   2> 386000 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.a.s.c.SolrXmlConfig MBean server found: 
com.sun.jmx.mbeanserver.JmxMBeanServer@14bf4aab, but no JMX reporters were 
configured - adding default JMX reporter.
   [junit4]   2> 386006 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51040/solr
   [junit4]   2> 386017 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) 
[n:127.0.0.1:39054_io%2Fhw    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 386020 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) 
[n:127.0.0.1:39054_io%2Fhw    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 386022 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) 
[n:127.0.0.1:39054_io%2Fhw    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:39054_io%2Fhw
   [junit4]   2> 386024 INFO  
(zkCallback-1215-thread-1-processing-n:127.0.0.1:40256_io%2Fhw) 
[n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (1) -> (2)
   [junit4]   2> 386030 INFO  
(zkCallback-1221-thread-1-processing-n:127.0.0.1:39054_io%2Fhw) 
[n:127.0.0.1:39054_io%2Fhw    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (1) -> (2)
   [junit4]   2> 386139 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) 
[n:127.0.0.1:39054_io%2Fhw    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@14bf4aab
   [junit4]   2> 386145 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) 
[n:127.0.0.1:39054_io%2Fhw    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@14bf4aab
   [junit4]   2> 386146 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) 
[n:127.0.0.1:39054_io%2Fhw    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@14bf4aab
   [junit4]   2> 386147 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) 
[n:127.0.0.1:39054_io%2Fhw    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrReplicationHandlerTest_B0D1D400E98C113B-001/jetty-002/cores
   [junit4]   2> 386178 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 386179 INFO  
(TEST-CdcrReplicationHandlerTest.testFullReplication-seed#[B0D1D400E98C113B]) [ 
   ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:51040/solr 
ready
   [junit4]   2> 386213 INFO  (qtp454697609-4200) [n:127.0.0.1:40256_io%2Fhw    
] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params 
pullReplicas=0&replicationFactor=1&name=tmp_collection&nrtReplicas=1&action=CREATE&numShards=1&tlogReplicas=0&createNodeSet=&wt=javabin&version=2
 and sendToOCPQueue=true
   [junit4]   2> 386216 INFO  
(OverseerThreadFactory-1108-thread-1-processing-n:127.0.0.1:40256_io%2Fhw) 
[n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.CreateCollectionCmd Create collection 
tmp_collection
   [junit4]   2> 386216 INFO  
(OverseerThreadFactory-1108-thread-1-processing-n:127.0.0.1:40256_io%2Fhw) 
[n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.CreateCollectionCmd Only one config set 
found in zk - using it:conf1
   [junit4]   2> 386217 WARN  
(OverseerThreadFactory-1108-thread-1-processing-n:127.0.0.1:40256_io%2Fhw) 
[n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.CreateCollectionCmd It is unusual to 
create a collection (tmp_collection) without cores.
   [junit4]   2> 386326 INFO  (qtp454697609-4200) [n:127.0.0.1:40256_io%2Fhw    
] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 
30 seconds. Check all shard replicas
   [junit4]   2> 386328 INFO  (qtp454697609-4200) [n:127.0.0.1:40256_io%2Fhw    
] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={pullReplicas=0&replicationFactor=1&name=tmp_collection&nrtReplicas=1&action=CREATE&numShards=1&tlogReplicas=0&createNodeSet=&wt=javabin&version=2}
 status=0 QTime=114
   [junit4]   2> 386330 INFO  (qtp454697609-4201) [n:127.0.0.1:40256_io%2Fhw    
] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with 
params 
node=127.0.0.1:40256_io%252Fhw&action=ADDREPLICA&collection=tmp_collection&shard=shard1&type=NRT&wt=javabin&version=2
 and sendToOCPQueue=true
   [junit4]   2> 386337 INFO  
(OverseerThreadFactory-1108-thread-2-processing-n:127.0.0.1:40256_io%2Fhw) 
[n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.AddReplicaCmd addReplica() : {
   [junit4]   2>   "collection":"tmp_collection",
   [junit4]   2>   "node":"127.0.0.1:40256_io%2Fhw",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "operation":"addreplica"}
   [junit4]   2> 386337 INFO  
(OverseerCollectionConfigSetProcessor-98191618537226245-127.0.0.1:40256_io%2Fhw-n_0000000000)
 [n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.OverseerTaskQueue Response ZK path: 
/overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may 
have disconnected from ZooKeeper
   [junit4]   2> 386341 INFO  
(OverseerThreadFactory-1108-thread-2-processing-n:127.0.0.1:40256_io%2Fhw) 
[n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.AddReplicaCmd Node Identified 
127.0.0.1:40256_io%2Fhw for creating new replica
   [junit4]   2> 386352 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw    
] o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&collection.configName=conf1&name=tmp_collection_shard1_replica_n0&action=CREATE&collection=tmp_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 386352 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw    
] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 
transient cores
   [junit4]   2> 386462 INFO  
(zkCallback-1215-thread-1-processing-n:127.0.0.1:40256_io%2Fhw) 
[n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/tmp_collection/state.json] for collection [tmp_collection] 
has occurred - updating... (live nodes size: [2])
   [junit4]   2> 387372 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 387386 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] 
o.a.s.s.IndexSchema [tmp_collection_shard1_replica_n0] Schema name=test
   [junit4]   2> 387494 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] 
o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 387515 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] 
o.a.s.c.CoreContainer Creating SolrCore 'tmp_collection_shard1_replica_n0' 
using configuration from collection tmp_collection, trusted=true
   [junit4]   2> 387516 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.core.tmp_collection.shard1.replica_n0' (registry 
'solr.core.tmp_collection.shard1.replica_n0') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@14bf4aab
   [junit4]   2> 387523 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] o.a.s.c.SolrCore 
solr.RecoveryStrategy.Builder
   [junit4]   2> 387523 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] o.a.s.c.SolrCore 
[[tmp_collection_shard1_replica_n0] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrReplicationHandlerTest_B0D1D400E98C113B-001/jetty-001/cores/tmp_collection_shard1_replica_n0],
 
dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrReplicationHandlerTest_B0D1D400E98C113B-001/jetty-001/cores/tmp_collection_shard1_replica_n0/data/]
   [junit4]   2> 387527 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.MockRandomMergePolicy: 
org.apache.lucene.index.MockRandomMergePolicy@11cb7e9a
   [junit4]   2> 387637 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] 
o.a.s.u.UpdateHandler Using UpdateLog implementation: 
org.apache.solr.update.CdcrUpdateLog
   [junit4]   2> 387638 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] 
o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 387639 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] 
o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 387639 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] 
o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 387641 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: 
maxMergeAtOnce=17, maxMergeAtOnceExplicit=29, maxMergedSegmentMB=77.1005859375, 
floorSegmentMB=2.068359375, forceMergeDeletesPctAllowed=6.89125328250852, 
segmentsPerTier=10.0, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=0.8958366005586522
   [junit4]   2> 387642 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@26b2563[tmp_collection_shard1_replica_n0] main]
   [junit4]   2> 387643 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf1
   [junit4]   2> 387644 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 387645 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 387651 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] 
o.a.s.h.CdcrBufferStateManager Created znode 
/collections/tmp_collection/cdcr/state/buffer
   [junit4]   2> 387656 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] 
o.a.s.h.CdcrProcessStateManager Created znode 
/collections/tmp_collection/cdcr/state/process
   [junit4]   2> 387674 INFO  
(searcherExecutor-1117-thread-1-processing-n:127.0.0.1:40256_io%2Fhw 
x:tmp_collection_shard1_replica_n0 s:shard1 c:tmp_collection) 
[n:127.0.0.1:40256_io%2Fhw c:tmp_collection s:shard1  
x:tmp_collection_shard1_replica_n0] o.a.s.c.SolrCore 
[tmp_collection_shard1_replica_n0] Registered new searcher 
Searcher@26b2563[tmp_collection_shard1_replica_n0] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 387674 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] 
o.a.s.u.UpdateLog Could not find max version in index or recent updates, using 
new clock 1571065899105910784
   [junit4]   2> 387681 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] 
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 387681 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] 
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 387681 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] 
o.a.s.c.SyncStrategy Sync replicas to 
https://127.0.0.1:40256/io/hw/tmp_collection_shard1_replica_n0/
   [junit4]   2> 387681 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] 
o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 387681 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] 
o.a.s.c.SyncStrategy 
https://127.0.0.1:40256/io/hw/tmp_collection_shard1_replica_n0/ has no replicas
   [junit4]   2> 387681 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] 
o.a.s.c.ShardLeaderElectionContext Found all replicas participating in 
election, clear LIR
   [junit4]   2> 387685 INFO  
(zkCallback-1215-thread-1-processing-n:127.0.0.1:40256_io%2Fhw) 
[n:127.0.0.1:40256_io%2Fhw    ] o.a.s.h.CdcrLeaderStateManager Received new 
leader state @ tmp_collection:shard1
   [junit4]   2> 387689 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] 
o.a.s.c.ShardLeaderElectionContext I am the new leader: 
https://127.0.0.1:40256/io/hw/tmp_collection_shard1_replica_n0/ shard1
   [junit4]   2> 387792 INFO  
(zkCallback-1215-thread-1-processing-n:127.0.0.1:40256_io%2Fhw) 
[n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/tmp_collection/state.json] for collection [tmp_collection] 
has occurred - updating... (live nodes size: [2])
   [junit4]   2> 387840 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] 
o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 387843 INFO  (qtp454697609-4199) [n:127.0.0.1:40256_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n0] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&collection.configName=conf1&name=tmp_collection_shard1_replica_n0&action=CREATE&collection=tmp_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT}
 status=0 QTime=1491
   [junit4]   2> 387847 INFO  (qtp454697609-4201) [n:127.0.0.1:40256_io%2Fhw    
] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={node=127.0.0.1:40256_io%252Fhw&action=ADDREPLICA&collection=tmp_collection&shard=shard1&type=NRT&wt=javabin&version=2}
 status=0 QTime=1516
   [junit4]   2> 387848 INFO  (qtp454697609-4194) [n:127.0.0.1:40256_io%2Fhw    
] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with 
params 
node=127.0.0.1:39054_io%252Fhw&action=ADDREPLICA&collection=tmp_collection&shard=shard1&type=NRT&wt=javabin&version=2
 and sendToOCPQueue=true
   [junit4]   2> 387853 INFO  
(OverseerCollectionConfigSetProcessor-98191618537226245-127.0.0.1:40256_io%2Fhw-n_0000000000)
 [n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.OverseerTaskQueue Response ZK path: 
/overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may 
have disconnected from ZooKeeper
   [junit4]   2> 387854 INFO  
(OverseerThreadFactory-1108-thread-3-processing-n:127.0.0.1:40256_io%2Fhw) 
[n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.AddReplicaCmd addReplica() : {
   [junit4]   2>   "collection":"tmp_collection",
   [junit4]   2>   "node":"127.0.0.1:39054_io%2Fhw",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "operation":"addreplica"}
   [junit4]   2> 387855 INFO  
(OverseerThreadFactory-1108-thread-3-processing-n:127.0.0.1:40256_io%2Fhw) 
[n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.AddReplicaCmd Node Identified 
127.0.0.1:39054_io%2Fhw for creating new replica
   [junit4]   2> 387908 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw    ] 
o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&collection.configName=conf1&name=tmp_collection_shard1_replica_n1&action=CREATE&collection=tmp_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 387909 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw    ] 
o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 
transient cores
   [junit4]   2> 388013 INFO  
(zkCallback-1215-thread-1-processing-n:127.0.0.1:40256_io%2Fhw) 
[n:127.0.0.1:40256_io%2Fhw    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/tmp_collection/state.json] for collection [tmp_collection] 
has occurred - updating... (live nodes size: [2])
   [junit4]   2> 388013 INFO  
(zkCallback-1221-thread-1-processing-n:127.0.0.1:39054_io%2Fhw) 
[n:127.0.0.1:39054_io%2Fhw    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/tmp_collection/state.json] for collection [tmp_collection] 
has occurred - updating... (live nodes size: [2])
   [junit4]   2> 388928 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 388939 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] 
o.a.s.s.IndexSchema [tmp_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 389034 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] 
o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 389062 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] 
o.a.s.c.CoreContainer Creating SolrCore 'tmp_collection_shard1_replica_n1' 
using configuration from collection tmp_collection, trusted=true
   [junit4]   2> 389062 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.core.tmp_collection.shard1.replica_n1' (registry 
'solr.core.tmp_collection.shard1.replica_n1') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@14bf4aab
   [junit4]   2> 389062 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] o.a.s.c.SolrCore 
solr.RecoveryStrategy.Builder
   [junit4]   2> 389063 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] o.a.s.c.SolrCore 
[[tmp_collection_shard1_replica_n1] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrReplicationHandlerTest_B0D1D400E98C113B-001/jetty-002/cores/tmp_collection_shard1_replica_n1],
 
dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrReplicationHandlerTest_B0D1D400E98C113B-001/jetty-002/cores/tmp_collection_shard1_replica_n1/data/]
   [junit4]   2> 389067 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.MockRandomMergePolicy: 
org.apache.lucene.index.MockRandomMergePolicy@290728fb
   [junit4]   2> 389132 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] 
o.a.s.u.UpdateHandler Using UpdateLog implementation: 
org.apache.solr.update.CdcrUpdateLog
   [junit4]   2> 389132 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] 
o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 389133 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] 
o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 389133 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] 
o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 389135 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: 
maxMergeAtOnce=17, maxMergeAtOnceExplicit=29, maxMergedSegmentMB=77.1005859375, 
floorSegmentMB=2.068359375, forceMergeDeletesPctAllowed=6.89125328250852, 
segmentsPerTier=10.0, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=0.8958366005586522
   [junit4]   2> 389135 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@5b1db555[tmp_collection_shard1_replica_n1] main]
   [junit4]   2> 389137 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf1
   [junit4]   2> 389137 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 389138 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 389146 INFO  
(searcherExecutor-1123-thread-1-processing-n:127.0.0.1:39054_io%2Fhw 
x:tmp_collection_shard1_replica_n1 s:shard1 c:tmp_collection) 
[n:127.0.0.1:39054_io%2Fhw c:tmp_collection s:shard1  
x:tmp_collection_shard1_replica_n1] o.a.s.c.SolrCore 
[tmp_collection_shard1_replica_n1] Registered new searcher 
Searcher@5b1db555[tmp_collection_shard1_replica_n1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 389146 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] 
o.a.s.u.UpdateLog Could not find max version in index or recent updates, using 
new clock 1571065900649414656
   [junit4]   2> 389151 INFO  (qtp454697609-4197) [n:127.0.0.1:40256_io%2Fhw 
c:tmp_collection s:shard1 r:core_node1 x:tmp_collection_shard1_replica_n0] 
o.a.s.c.S.Request [tmp_collection_shard1_replica_n0]  webapp=/io/hw path=/cdcr 
params={action=LASTPROCESSEDVERSION&wt=javabin&version=2} status=0 QTime=1
   [junit4]   2> 389156 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] 
o.a.s.c.ZkController Core needs to recover:tmp_collection_shard1_replica_n1
   [junit4]   2> 389157 INFO  
(updateExecutor-1218-thread-1-processing-n:127.0.0.1:39054_io%2Fhw 
x:tmp_collection_shard1_replica_n1 s:shard1 c:tmp_collection) 
[n:127.0.0.1:39054_io%2Fhw c:tmp_collection s:shard1 r:core_node2 
x:tmp_collection_shard1_replica_n1] o.a.s.u.DefaultSolrCoreState Running 
recovery
   [junit4]   2> 389157 INFO  
(recoveryExecutor-1219-thread-1-processing-n:127.0.0.1:39054_io%2Fhw 
x:tmp_collection_shard1_replica_n1 s:shard1 c:tmp_collection r:core_node2) 
[n:127.0.0.1:39054_io%2Fhw c:tmp_collection s:shard1 r:core_node2 
x:tmp_collection_shard1_replica_n1] o.a.s.c.RecoveryStrategy Starting recovery 
process. recoveringAfterStartup=true
   [junit4]   2> 389158 INFO  
(recoveryExecutor-1219-thread-1-processing-n:127.0.0.1:39054_io%2Fhw 
x:tmp_collection_shard1_replica_n1 s:shard1 c:tmp_collection r:core_node2) 
[n:127.0.0.1:39054_io%2Fhw c:tmp_collection s:shard1 r:core_node2 
x:tmp_collection_shard1_replica_n1] o.a.s.c.RecoveryStrategy ###### 
startupVersions=[[]]
   [junit4]   2> 389158 INFO  
(recoveryExecutor-1219-thread-1-processing-n:127.0.0.1:39054_io%2Fhw 
x:tmp_collection_shard1_replica_n1 s:shard1 c:tmp_collection r:core_node2) 
[n:127.0.0.1:39054_io%2Fhw c:tmp_collection s:shard1 r:core_node2 
x:tmp_collection_shard1_replica_n1] o.a.s.c.RecoveryStrategy Begin buffering 
updates. core=[tmp_collection_shard1_replica_n1]
   [junit4]   2> 389158 INFO  
(recoveryExecutor-1219-thread-1-processing-n:127.0.0.1:39054_io%2Fhw 
x:tmp_collection_shard1_replica_n1 s:shard1 c:tmp_collection r:core_node2) 
[n:127.0.0.1:39054_io%2Fhw c:tmp_collection s:shard1 r:core_node2 
x:tmp_collection_shard1_replica_n1] o.a.s.u.UpdateLog Starting to buffer 
updates. FSUpdateLog{state=ACTIVE, tlog=null}
   [junit4]   2> 389158 INFO  
(recoveryExecutor-1219-thread-1-processing-n:127.0.0.1:39054_io%2Fhw 
x:tmp_collection_shard1_replica_n1 s:shard1 c:tmp_collection r:core_node2) 
[n:127.0.0.1:39054_io%2Fhw c:tmp_collection s:shard1 r:core_node2 
x:tmp_collection_shard1_replica_n1] o.a.s.c.RecoveryStrategy Publishing state 
of core [tmp_collection_shard1_replica_n1] as recovering, leader is 
[https://127.0.0.1:40256/io/hw/tmp_collection_shard1_replica_n0/] and I am 
[https://127.0.0.1:39054/io/hw/tmp_collection_shard1_replica_n1/]
   [junit4]   2> 389158 INFO  (qtp8146820-4226) [n:127.0.0.1:39054_io%2Fhw 
c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n1] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&collection.configName=conf1&name=tmp_collection_shard1_replica_n1&action=CREATE&collection=tmp_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT}
 status=0 QTime=1250
   [junit4]   2> 389164 INFO  (qtp454697609-4194) [n:127.0.0.1:40256_io%2Fhw    
] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={node=127.0.0.1:39054_io%252Fhw&action=ADDREPLICA&collection=tmp_collection&shard=shard1&type=NRT&wt=javabin&version=2}
 status=0 QTime=1316
   [junit4]   2> 389164 INFO  
(recoveryExecutor-1219-thread-1-processing-n:127.0.0.1:39054_io%2Fhw 
x:tmp_collection_shard1_replica_n1 s:shard1 c:tmp_collection r:core_node2) 
[n:127.0.0.1:39054_io%2Fhw c:tmp_collection s:shard1 r:core_node2 
x:tmp_collection_shard1_replica_n1] o.a.s.c.RecoveryStrategy Sending prep 
recovery command to [https://127.0.0.1:40256/io/hw]; [WaitForState: 
action=PREPRECOVERY&core=tmp_collection_shard1_replica_n0&nodeName=127.0.0.1:39054_io%252Fhw&coreNodeName=core_node2&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
   [junit4]   2> 389167 INFO  (qtp454697609-4200) [n:127.0.0.1:40256_io%2Fhw    
] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node2, state: 
recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true, 
maxTime: 183 s
   [junit4]   2> 389171 INFO  (qtp454697609-4200) [n:127.0.0.1:40256_io%2Fhw    
] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): 
collection=tmp_collection, shard=shard1, 
thisCore=tmp_collection_shard1_replica_n0, leaderDoesNotNeedRecovery=false, 
isLeader? true, live=true, checkLive=true, currentState=down, 
localState=active, nodeName=127.0.0.1:39054_io%2Fhw, coreNodeName=core_node2, 
onlyIfActiveCheckResult=false, nodeProps: 
core_node2:{"core":"tmp_collection_shard1_replica_n1","base_url":"https://127.0.0.1:39054/io/hw","node_name":"127.0.0.1:39054_io%2Fhw","state":"down","type":"NRT"}
   [junit4]   1> / (2)
   [junit4]   1> DATA:
   [junit4]   1>     
   [junit4]   1>  /zookeeper (1)
   [junit4]   1>  DATA:
   [junit4]   1>      
   [junit4]   1>  /solr (10)
   [junit4]   1>   /solr/configs (1)
   [junit4]   1>    /solr/configs/conf1 (11)
   [junit4]   1>     /solr/configs/conf1/currency.xml (0)
   [junit4]   1>     DATA: ...supressed...
   [junit4]   1>     /solr/configs/conf1/enumsConfig.xml (0)
   [junit4]   1>     DATA: ...s

[...truncated too long message...]

 instance=1045063570
   [junit4]   2> 27211 INFO  (jetty-closer-24-thread-2) [    ] 
o.a.s.c.CoreContainer Shutting down CoreContainer instance=1746893512
   [junit4]   2> 27217 INFO  (jetty-closer-24-thread-1) [    ] 
o.a.s.c.CoreContainer Shutting down CoreContainer instance=1460563256
   [junit4]   2> 27218 INFO  (jetty-closer-24-thread-2) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, 
tag=null
   [junit4]   2> 27218 INFO  (jetty-closer-24-thread-3) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, 
tag=null
   [junit4]   2> 27218 INFO  (jetty-closer-24-thread-1) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, 
tag=null
   [junit4]   2> 27224 INFO  (jetty-closer-24-thread-3) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, 
tag=null
   [junit4]   2> 27226 INFO  (jetty-closer-24-thread-2) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, 
tag=null
   [junit4]   2> 27229 INFO  (jetty-closer-24-thread-1) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, 
tag=null
   [junit4]   2> 27233 INFO  (jetty-closer-24-thread-3) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, 
tag=null
   [junit4]   2> 27235 INFO  (jetty-closer-24-thread-2) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, 
tag=null
   [junit4]   2> 27236 INFO  (jetty-closer-24-thread-1) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, 
tag=null
   [junit4]   2> 27238 INFO  (jetty-closer-24-thread-3) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, 
tag=null
   [junit4]   2> 27239 INFO  (jetty-closer-24-thread-3) [    ] 
o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 
transient cores
   [junit4]   2> 27239 INFO  (jetty-closer-24-thread-2) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, 
tag=null
   [junit4]   2> 27242 INFO  (jetty-closer-24-thread-3) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@268c9ce3{/solr,null,UNAVAILABLE}
   [junit4]   2> 27242 INFO  (jetty-closer-24-thread-2) [    ] 
o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 
transient cores
   [junit4]   2> 27242 INFO  (jetty-closer-24-thread-1) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, 
tag=null
   [junit4]   2> 27243 INFO  
(zkCallback-38-thread-2-processing-n:127.0.0.1:49345_solr) 
[n:127.0.0.1:49345_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (3) -> (2)
   [junit4]   2> 27243 INFO  
(zkCallback-40-thread-2-processing-n:127.0.0.1:50282_solr) 
[n:127.0.0.1:50282_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (3) -> (2)
   [junit4]   2> 27245 INFO  (coreCloseExecutor-178-thread-1) 
[n:127.0.0.1:49345_solr c:collection1 s:shard1 r:core_node1 
x:collection1_shard1_replica_n1] o.a.s.c.SolrCore 
[collection1_shard1_replica_n1]  CLOSING SolrCore 
org.apache.solr.core.SolrCore@1c753d9a
   [junit4]   2> 27245 INFO  (jetty-closer-24-thread-2) [    ] o.a.s.c.Overseer 
Overseer (id=98191805058580489-127.0.0.1:50282_solr-n_0000000000) closing
   [junit4]   2> 27245 INFO  
(OverseerStateUpdate-98191805058580489-127.0.0.1:50282_solr-n_0000000000) 
[n:127.0.0.1:50282_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 
127.0.0.1:50282_solr
   [junit4]   2> 27254 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from 
client sessionid 0x15cd8ed2c5f0009, likely client has closed socket
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 27254 INFO  
(zkCallback-38-thread-1-processing-n:127.0.0.1:49345_solr) 
[n:127.0.0.1:49345_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (2) -> (1)
   [junit4]   2> 27255 INFO  
(zkCallback-38-thread-2-processing-n:127.0.0.1:49345_solr) 
[n:127.0.0.1:49345_solr    ] o.a.s.c.OverseerElectionContext I am going to be 
the leader 127.0.0.1:49345_solr
   [junit4]   2> 27261 INFO  (coreCloseExecutor-178-thread-1) 
[n:127.0.0.1:49345_solr c:collection1 s:shard1 r:core_node1 
x:collection1_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric 
reporters for registry=solr.core.collection1.shard1.replica_n1, tag=477445530
   [junit4]   2> 27269 INFO  (coreCloseExecutor-178-thread-1) 
[n:127.0.0.1:49345_solr c:collection1 s:shard1 r:core_node1 
x:collection1_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric 
reporters for registry=solr.collection.collection1.shard1.leader, tag=477445530
   [junit4]   2> 27281 WARN  
(zkCallback-38-thread-2-processing-n:127.0.0.1:49345_solr) 
[n:127.0.0.1:49345_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, 
but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 27281 INFO  (jetty-closer-24-thread-1) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@43541600{/solr,null,UNAVAILABLE}
   [junit4]   2> 28753 WARN  
(zkCallback-40-thread-2-processing-n:127.0.0.1:50282_solr) 
[n:127.0.0.1:50282_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, 
but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 28754 INFO  (jetty-closer-24-thread-2) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@4213cf1e{/solr,null,UNAVAILABLE}
   [junit4]   2> 28755 ERROR 
(TEST-TestLTROnSolrCloud.testSimpleQuery-seed#[C7CE671D17FFDB46]) [    ] 
o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper 
server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 28757 INFO  
(TEST-TestLTROnSolrCloud.testSimpleQuery-seed#[C7CE671D17FFDB46]) [    ] 
o.a.s.c.ZkTestServer connecting to 127.0.0.1:43036 43036
   [junit4]   2> 28860 INFO  (Thread-57) [    ] o.a.s.c.ZkTestServer connecting 
to 127.0.0.1:43036 43036
   [junit4]   2> 28863 WARN  (Thread-57) [    ] o.a.s.c.ZkTestServer Watch 
limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2>        5       /solr/aliases.json
   [junit4]   2>        5       /solr/clusterprops.json
   [junit4]   2>        4       /solr/security.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2>        5       /solr/clusterstate.json
   [junit4]   2>        5       /solr/collections/collection1/state.json
   [junit4]   2>        2       
/solr/overseer_elect/election/98191805058580489-127.0.0.1:50282_solr-n_0000000000
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2>        5       /solr/live_nodes
   [junit4]   2>        5       /solr/collections
   [junit4]   2> 
   [junit4]   2> 28863 INFO  
(TEST-TestLTROnSolrCloud.testSimpleQuery-seed#[C7CE671D17FFDB46]) [    ] 
o.a.s.SolrTestCaseJ4 ###Ending testSimpleQuery
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 
'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestLTROnSolrCloud 
-Dtests.method=testSimpleQuery -Dtests.seed=C7CE671D17FFDB46 
-Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt
 -Dtests.locale=ar-IQ -Dtests.timezone=America/Asuncion -Dtests.asserts=true 
-Dtests.file.encoding=US-ASCII
   [junit4] ERROR   9.01s J0 | TestLTROnSolrCloud.testSimpleQuery <<<
   [junit4]    > Throwable #1: java.lang.RuntimeException: Path not found: 
/responseHeader/status
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([C7CE671D17FFDB46:B2C3A70107E51D93]:0)
   [junit4]    >        at 
org.apache.solr.util.RestTestBase.assertJPut(RestTestBase.java:329)
   [junit4]    >        at 
org.apache.solr.util.RestTestBase.assertJPut(RestTestBase.java:271)
   [junit4]    >        at 
org.apache.solr.ltr.TestRerankBase.loadFeature(TestRerankBase.java:286)
   [junit4]    >        at 
org.apache.solr.ltr.TestLTROnSolrCloud.loadModelsAndFeatures(TestLTROnSolrCloud.java:184)
   [junit4]    >        at 
org.apache.solr.ltr.TestLTROnSolrCloud.setupSolrCluster(TestLTROnSolrCloud.java:138)
   [junit4]    >        at 
org.apache.solr.ltr.TestLTROnSolrCloud.setUp(TestLTROnSolrCloud.java:58)
   [junit4]    >        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/contrib/solr-ltr/test/J0/temp/solr.ltr.TestLTROnSolrCloud_C7CE671D17FFDB46-001
   [junit4]   2> Jun 24, 2017 7:07:29 AM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked 
thread(s).
   [junit4]   2> NOTE: test params are: 
codec=DummyCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=DUMMY,
 chunkSize=13190, maxDocsPerChunk=5, blockSize=4), 
termVectorsFormat=CompressingTermVectorsFormat(compressionMode=DUMMY, 
chunkSize=13190, blockSize=4)), sim=RandomSimilarity(queryNorm=false): {}, 
locale=ar-IQ, timezone=America/Asuncion
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 
1.8.0_131 (64-bit)/cpus=4,threads=1,free=166008960,total=337117184
   [junit4]   2> NOTE: All tests run in this JVM: [TestRankingFeature, 
TestExternalValueFeatures, TestUserTermScorereQDF, TestFieldLengthFeature, 
TestManagedFeatureStore, TestNoMatchSolrFeature, TestLTROnSolrCloud]
   [junit4] Completed [29/33 (1!)] on J0 in 9.31s, 1 test, 1 error <<< FAILURES!

[...truncated 36772 lines...]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to