Build: https://builds.apache.org/job/Lucene-Solr-Tests-master/2327/

2 tests failed.
FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.DeleteInactiveReplicaTest

Error Message:
1 thread leaked from SUITE scope at 
org.apache.solr.cloud.DeleteInactiveReplicaTest:     1) Thread[id=17650, 
name=qtp2081593016-17650, state=TIMED_WAITING, 
group=TGRP-DeleteInactiveReplicaTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) 
        at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE 
scope at org.apache.solr.cloud.DeleteInactiveReplicaTest: 
   1) Thread[id=17650, name=qtp2081593016-17650, state=TIMED_WAITING, 
group=TGRP-DeleteInactiveReplicaTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
        at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
        at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([8A256E9AC754A98D]:0)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.DeleteInactiveReplicaTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) 
Thread[id=17650, name=qtp2081593016-17650, state=TIMED_WAITING, 
group=TGRP-DeleteInactiveReplicaTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) 
        at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie 
threads that couldn't be terminated:
   1) Thread[id=17650, name=qtp2081593016-17650, state=TIMED_WAITING, 
group=TGRP-DeleteInactiveReplicaTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
        at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
        at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([8A256E9AC754A98D]:0)




Build Log:
[...truncated 13685 lines...]
   [junit4] Suite: org.apache.solr.cloud.DeleteInactiveReplicaTest
   [junit4]   2> 1746657 INFO  
(SUITE-DeleteInactiveReplicaTest-seed#[8A256E9AC754A98D]-worker) [    ] 
o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: 
test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteInactiveReplicaTest_8A256E9AC754A98D-001/init-core-data-001
   [junit4]   2> 1746657 WARN  
(SUITE-DeleteInactiveReplicaTest-seed#[8A256E9AC754A98D]-worker) [    ] 
o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=7 numCloses=7
   [junit4]   2> 1746657 INFO  
(SUITE-DeleteInactiveReplicaTest-seed#[8A256E9AC754A98D]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) 
w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 1746659 INFO  
(SUITE-DeleteInactiveReplicaTest-seed#[8A256E9AC754A98D]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: 
@org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 1746660 INFO  
(SUITE-DeleteInactiveReplicaTest-seed#[8A256E9AC754A98D]-worker) [    ] 
o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteInactiveReplicaTest_8A256E9AC754A98D-001/tempDir-001
   [junit4]   2> 1746661 INFO  
(SUITE-DeleteInactiveReplicaTest-seed#[8A256E9AC754A98D]-worker) [    ] 
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1746661 INFO  (Thread-4059) [    ] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1746661 INFO  (Thread-4059) [    ] o.a.s.c.ZkTestServer 
Starting server
   [junit4]   2> 1746662 ERROR (Thread-4059) [    ] o.a.z.s.ZooKeeperServer 
ZKShutdownHandler is not registered, so ZooKeeper server won't take any action 
on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1746761 INFO  
(SUITE-DeleteInactiveReplicaTest-seed#[8A256E9AC754A98D]-worker) [    ] 
o.a.s.c.ZkTestServer start zk server on port:34083
   [junit4]   2> 1746763 INFO  (zkConnectionManagerCallback-3779-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1746769 INFO  (jetty-launcher-3776-thread-1) [    ] 
o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 
2017-11-21T16:27:37-05:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1746770 INFO  (jetty-launcher-3776-thread-2) [    ] 
o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 
2017-11-21T16:27:37-05:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1746772 INFO  (jetty-launcher-3776-thread-3) [    ] 
o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 
2017-11-21T16:27:37-05:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1746774 INFO  (jetty-launcher-3776-thread-4) [    ] 
o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 
2017-11-21T16:27:37-05:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1746775 INFO  (jetty-launcher-3776-thread-3) [    ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1746775 INFO  (jetty-launcher-3776-thread-3) [    ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1746775 INFO  (jetty-launcher-3776-thread-3) [    ] 
o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 1746775 INFO  (jetty-launcher-3776-thread-3) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@18d64fd6{/solr,null,AVAILABLE}
   [junit4]   2> 1746776 INFO  (jetty-launcher-3776-thread-4) [    ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1746776 INFO  (jetty-launcher-3776-thread-4) [    ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1746776 INFO  (jetty-launcher-3776-thread-4) [    ] 
o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 1746776 INFO  (jetty-launcher-3776-thread-4) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@49d138ba{/solr,null,AVAILABLE}
   [junit4]   2> 1746776 INFO  (jetty-launcher-3776-thread-3) [    ] 
o.e.j.s.AbstractConnector Started ServerConnector@7b0f3b20{SSL,[ssl, 
http/1.1]}{127.0.0.1:57581}
   [junit4]   2> 1746776 INFO  (jetty-launcher-3776-thread-3) [    ] 
o.e.j.s.Server Started @1749055ms
   [junit4]   2> 1746776 INFO  (jetty-launcher-3776-thread-3) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=57581}
   [junit4]   2> 1746776 INFO  (jetty-launcher-3776-thread-4) [    ] 
o.e.j.s.AbstractConnector Started ServerConnector@32747ad9{SSL,[ssl, 
http/1.1]}{127.0.0.1:34551}
   [junit4]   2> 1746777 INFO  (jetty-launcher-3776-thread-4) [    ] 
o.e.j.s.Server Started @1749056ms
   [junit4]   2> 1746777 INFO  (jetty-launcher-3776-thread-4) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=34551}
   [junit4]   2> 1746777 ERROR (jetty-launcher-3776-thread-3) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 1746777 ERROR (jetty-launcher-3776-thread-4) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 1746777 INFO  (jetty-launcher-3776-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 
8.0.0
   [junit4]   2> 1746777 INFO  (jetty-launcher-3776-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 
8.0.0
   [junit4]   2> 1746777 INFO  (jetty-launcher-3776-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1746777 INFO  (jetty-launcher-3776-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1746777 INFO  (jetty-launcher-3776-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1746777 INFO  (jetty-launcher-3776-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2018-02-14T21:58:36.320Z
   [junit4]   2> 1746777 INFO  (jetty-launcher-3776-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1746778 INFO  (jetty-launcher-3776-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2018-02-14T21:58:36.321Z
   [junit4]   2> 1746783 INFO  (jetty-launcher-3776-thread-2) [    ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1746783 INFO  (jetty-launcher-3776-thread-2) [    ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1746783 INFO  (jetty-launcher-3776-thread-2) [    ] 
o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 1746787 INFO  (jetty-launcher-3776-thread-1) [    ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1746787 INFO  (jetty-launcher-3776-thread-1) [    ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1746787 INFO  (jetty-launcher-3776-thread-1) [    ] 
o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 1746787 INFO  (jetty-launcher-3776-thread-1) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@2005ed32{/solr,null,AVAILABLE}
   [junit4]   2> 1746788 INFO  (jetty-launcher-3776-thread-1) [    ] 
o.e.j.s.AbstractConnector Started ServerConnector@6d1047a3{SSL,[ssl, 
http/1.1]}{127.0.0.1:42420}
   [junit4]   2> 1746788 INFO  (jetty-launcher-3776-thread-1) [    ] 
o.e.j.s.Server Started @1749067ms
   [junit4]   2> 1746788 INFO  (jetty-launcher-3776-thread-1) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=42420}
   [junit4]   2> 1746788 ERROR (jetty-launcher-3776-thread-1) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 1746788 INFO  (jetty-launcher-3776-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 
8.0.0
   [junit4]   2> 1746788 INFO  (jetty-launcher-3776-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1746788 INFO  (jetty-launcher-3776-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1746789 INFO  (jetty-launcher-3776-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2018-02-14T21:58:36.332Z
   [junit4]   2> 1746790 INFO  (jetty-launcher-3776-thread-2) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@35faa277{/solr,null,AVAILABLE}
   [junit4]   2> 1746790 INFO  (jetty-launcher-3776-thread-2) [    ] 
o.e.j.s.AbstractConnector Started ServerConnector@41c53295{SSL,[ssl, 
http/1.1]}{127.0.0.1:60333}
   [junit4]   2> 1746790 INFO  (jetty-launcher-3776-thread-2) [    ] 
o.e.j.s.Server Started @1749069ms
   [junit4]   2> 1746790 INFO  (jetty-launcher-3776-thread-2) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=60333}
   [junit4]   2> 1746791 ERROR (jetty-launcher-3776-thread-2) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 1746791 INFO  (jetty-launcher-3776-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 
8.0.0
   [junit4]   2> 1746791 INFO  (jetty-launcher-3776-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1746791 INFO  (jetty-launcher-3776-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1746791 INFO  (jetty-launcher-3776-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2018-02-14T21:58:36.334Z
   [junit4]   2> 1746793 INFO  (zkConnectionManagerCallback-3783-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1746793 INFO  (zkConnectionManagerCallback-3785-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1746793 INFO  (zkConnectionManagerCallback-3787-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1746795 INFO  (jetty-launcher-3776-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1746795 INFO  (jetty-launcher-3776-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1746795 INFO  (zkConnectionManagerCallback-3781-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1746795 INFO  (jetty-launcher-3776-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1746800 INFO  (jetty-launcher-3776-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1746805 INFO  (jetty-launcher-3776-thread-2) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34083/solr
   [junit4]   2> 1746805 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 
0x104665fcfc70004, likely client has closed socket
   [junit4]   2> 1746806 INFO  (jetty-launcher-3776-thread-1) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34083/solr
   [junit4]   2> 1746808 INFO  (zkConnectionManagerCallback-3795-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1746815 INFO  
(zkConnectionManagerCallback-3797-thread-1-processing-n:127.0.0.1:42420_solr) 
[n:127.0.0.1:42420_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1746816 INFO  (jetty-launcher-3776-thread-3) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34083/solr
   [junit4]   2> 1746817 INFO  (zkConnectionManagerCallback-3792-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1746819 INFO  (jetty-launcher-3776-thread-4) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34083/solr
   [junit4]   2> 1746820 INFO  (zkConnectionManagerCallback-3801-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1746821 INFO  (zkConnectionManagerCallback-3805-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1746822 INFO  
(zkConnectionManagerCallback-3807-thread-1-processing-n:127.0.0.1:60333_solr) 
[n:127.0.0.1:60333_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1746824 INFO  
(zkConnectionManagerCallback-3809-thread-1-processing-n:127.0.0.1:57581_solr) 
[n:127.0.0.1:57581_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1746893 INFO  
(zkConnectionManagerCallback-3812-thread-1-processing-n:127.0.0.1:34551_solr) 
[n:127.0.0.1:34551_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1746894 INFO  (jetty-launcher-3776-thread-1) 
[n:127.0.0.1:42420_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1746899 INFO  (jetty-launcher-3776-thread-1) 
[n:127.0.0.1:42420_solr    ] o.a.s.c.OverseerElectionContext I am going to be 
the leader 127.0.0.1:42420_solr
   [junit4]   2> 1746902 INFO  (jetty-launcher-3776-thread-1) 
[n:127.0.0.1:42420_solr    ] o.a.s.c.Overseer Overseer 
(id=73296055638622215-127.0.0.1:42420_solr-n_0000000000) starting
   [junit4]   2> 1746908 INFO  (jetty-launcher-3776-thread-4) 
[n:127.0.0.1:34551_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1746910 INFO  (jetty-launcher-3776-thread-4) 
[n:127.0.0.1:34551_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:34551_solr
   [junit4]   2> 1746919 INFO  
(zkCallback-3796-thread-1-processing-n:127.0.0.1:42420_solr) 
[n:127.0.0.1:42420_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 1746920 INFO  
(zkCallback-3811-thread-1-processing-n:127.0.0.1:34551_solr) 
[n:127.0.0.1:34551_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 1746923 INFO  (jetty-launcher-3776-thread-1) 
[n:127.0.0.1:42420_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:42420_solr
   [junit4]   2> 1746924 INFO  
(zkCallback-3811-thread-1-processing-n:127.0.0.1:34551_solr) 
[n:127.0.0.1:34551_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (1) -> (2)
   [junit4]   2> 1746927 INFO  
(zkCallback-3796-thread-1-processing-n:127.0.0.1:42420_solr) 
[n:127.0.0.1:42420_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (1) -> (2)
   [junit4]   2> 1746937 INFO  (jetty-launcher-3776-thread-3) 
[n:127.0.0.1:57581_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (2)
   [junit4]   2> 1746938 INFO  (jetty-launcher-3776-thread-3) 
[n:127.0.0.1:57581_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1746939 INFO  (jetty-launcher-3776-thread-3) 
[n:127.0.0.1:57581_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:57581_solr
   [junit4]   2> 1746940 INFO  
(zkCallback-3796-thread-1-processing-n:127.0.0.1:42420_solr) 
[n:127.0.0.1:42420_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (2) -> (3)
   [junit4]   2> 1746940 INFO  
(zkCallback-3811-thread-1-processing-n:127.0.0.1:34551_solr) 
[n:127.0.0.1:34551_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (2) -> (3)
   [junit4]   2> 1746951 INFO  
(zkCallback-3808-thread-1-processing-n:127.0.0.1:57581_solr) 
[n:127.0.0.1:57581_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (2) -> (3)
   [junit4]   2> 1746964 INFO  (jetty-launcher-3776-thread-2) 
[n:127.0.0.1:60333_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (3)
   [junit4]   2> 1746965 INFO  (jetty-launcher-3776-thread-2) 
[n:127.0.0.1:60333_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1746967 INFO  (jetty-launcher-3776-thread-2) 
[n:127.0.0.1:60333_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:60333_solr
   [junit4]   2> 1746974 INFO  
(zkCallback-3796-thread-1-processing-n:127.0.0.1:42420_solr) 
[n:127.0.0.1:42420_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (3) -> (4)
   [junit4]   2> 1746974 INFO  
(zkCallback-3811-thread-1-processing-n:127.0.0.1:34551_solr) 
[n:127.0.0.1:34551_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (3) -> (4)
   [junit4]   2> 1746995 INFO  
(zkCallback-3808-thread-1-processing-n:127.0.0.1:57581_solr) 
[n:127.0.0.1:57581_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (3) -> (4)
   [junit4]   2> 1746996 INFO  
(zkCallback-3806-thread-1-processing-n:127.0.0.1:60333_solr) 
[n:127.0.0.1:60333_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (3) -> (4)
   [junit4]   2> 1747002 INFO  (jetty-launcher-3776-thread-3) 
[n:127.0.0.1:57581_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_57581.solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@1c23a6f3
   [junit4]   2> 1747008 INFO  (jetty-launcher-3776-thread-3) 
[n:127.0.0.1:57581_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_57581.solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@1c23a6f3
   [junit4]   2> 1747009 INFO  (jetty-launcher-3776-thread-3) 
[n:127.0.0.1:57581_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_57581.solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@1c23a6f3
   [junit4]   2> 1747010 INFO  (jetty-launcher-3776-thread-3) 
[n:127.0.0.1:57581_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteInactiveReplicaTest_8A256E9AC754A98D-001/tempDir-001/node3/.
   [junit4]   2> 1747070 INFO  (jetty-launcher-3776-thread-2) 
[n:127.0.0.1:60333_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_60333.solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@1c23a6f3
   [junit4]   2> 1747078 INFO  (jetty-launcher-3776-thread-2) 
[n:127.0.0.1:60333_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_60333.solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@1c23a6f3
   [junit4]   2> 1747078 INFO  (jetty-launcher-3776-thread-2) 
[n:127.0.0.1:60333_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_60333.solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@1c23a6f3
   [junit4]   2> 1747080 INFO  (jetty-launcher-3776-thread-2) 
[n:127.0.0.1:60333_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteInactiveReplicaTest_8A256E9AC754A98D-001/tempDir-001/node2/.
   [junit4]   2> 1747125 INFO  (jetty-launcher-3776-thread-4) 
[n:127.0.0.1:34551_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_34551.solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@1c23a6f3
   [junit4]   2> 1747131 INFO  (jetty-launcher-3776-thread-1) 
[n:127.0.0.1:42420_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_42420.solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@1c23a6f3
   [junit4]   2> 1747133 INFO  (jetty-launcher-3776-thread-4) 
[n:127.0.0.1:34551_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_34551.solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@1c23a6f3
   [junit4]   2> 1747133 INFO  (jetty-launcher-3776-thread-4) 
[n:127.0.0.1:34551_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_34551.solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@1c23a6f3
   [junit4]   2> 1747134 INFO  (jetty-launcher-3776-thread-4) 
[n:127.0.0.1:34551_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteInactiveReplicaTest_8A256E9AC754A98D-001/tempDir-001/node4/.
   [junit4]   2> 1747147 INFO  (jetty-launcher-3776-thread-1) 
[n:127.0.0.1:42420_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_42420.solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@1c23a6f3
   [junit4]   2> 1747147 INFO  (jetty-launcher-3776-thread-1) 
[n:127.0.0.1:42420_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_42420.solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@1c23a6f3
   [junit4]   2> 1747148 INFO  (jetty-launcher-3776-thread-1) 
[n:127.0.0.1:42420_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteInactiveReplicaTest_8A256E9AC754A98D-001/tempDir-001/node1/.
   [junit4]   2> 1747231 INFO  (zkConnectionManagerCallback-3821-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1747240 INFO  (zkConnectionManagerCallback-3825-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1747241 INFO  
(SUITE-DeleteInactiveReplicaTest-seed#[8A256E9AC754A98D]-worker) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 1747242 INFO  
(SUITE-DeleteInactiveReplicaTest-seed#[8A256E9AC754A98D]-worker) [    ] 
o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34083/solr ready
   [junit4]   2> 1747270 INFO  
(TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[8A256E9AC754A98D])
 [    ] o.a.s.SolrTestCaseJ4 ###Starting deleteInactiveReplicaTest
   [junit4]   2> 1747288 INFO  (qtp840256473-17481) [n:127.0.0.1:34551_solr    
] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params 
replicationFactor=2&collection.configName=conf&maxShardsPerNode=2&name=delDeadColl&nrtReplicas=2&action=CREATE&numShards=2&wt=javabin&version=2
 and sendToOCPQueue=true
   [junit4]   2> 1747290 INFO  
(OverseerThreadFactory-8584-thread-1-processing-n:127.0.0.1:42420_solr) 
[n:127.0.0.1:42420_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection 
delDeadColl
   [junit4]   2> 1747396 INFO  
(OverseerStateUpdate-73296055638622215-127.0.0.1:42420_solr-n_0000000000) 
[n:127.0.0.1:42420_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"delDeadColl",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"delDeadColl_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:34551/solr";,
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 1747398 INFO  
(OverseerStateUpdate-73296055638622215-127.0.0.1:42420_solr-n_0000000000) 
[n:127.0.0.1:42420_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"delDeadColl",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"delDeadColl_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:42420/solr";,
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 1747400 INFO  
(OverseerStateUpdate-73296055638622215-127.0.0.1:42420_solr-n_0000000000) 
[n:127.0.0.1:42420_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"delDeadColl",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"delDeadColl_shard2_replica_n4",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:60333/solr";,
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 1747402 INFO  
(OverseerStateUpdate-73296055638622215-127.0.0.1:42420_solr-n_0000000000) 
[n:127.0.0.1:42420_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"delDeadColl",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"delDeadColl_shard2_replica_n6",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:57581/solr";,
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 1747617 INFO  (qtp840256473-17486) [n:127.0.0.1:34551_solr    
] o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=delDeadColl_shard1_replica_n1&action=CREATE&numShards=2&collection=delDeadColl&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1747617 INFO  (qtp840256473-17486) [n:127.0.0.1:34551_solr    
] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 
2147483647 transient cores
   [junit4]   2> 1747639 INFO  (qtp2081593016-17467) [n:127.0.0.1:60333_solr    
] o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&coreNodeName=core_node7&collection.configName=conf&newCollection=true&name=delDeadColl_shard2_replica_n4&action=CREATE&numShards=2&collection=delDeadColl&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1747644 INFO  (qtp2081593016-17467) [n:127.0.0.1:60333_solr    
] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 
2147483647 transient cores
   [junit4]   2> 1747649 INFO  (qtp603739474-17477) [n:127.0.0.1:57581_solr    
] o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&coreNodeName=core_node8&collection.configName=conf&newCollection=true&name=delDeadColl_shard2_replica_n6&action=CREATE&numShards=2&collection=delDeadColl&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1747649 INFO  (qtp603739474-17477) [n:127.0.0.1:57581_solr    
] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 
2147483647 transient cores
   [junit4]   2> 1747651 INFO  (qtp1934983229-17497) [n:127.0.0.1:42420_solr    
] o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf&newCollection=true&name=delDeadColl_shard1_replica_n2&action=CREATE&numShards=2&collection=delDeadColl&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1747651 INFO  (qtp1934983229-17497) [n:127.0.0.1:42420_solr    
] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 
2147483647 transient cores
   [junit4]   2> 1747757 INFO  
(zkCallback-3806-thread-1-processing-n:127.0.0.1:60333_solr) 
[n:127.0.0.1:60333_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1747757 INFO  
(zkCallback-3808-thread-1-processing-n:127.0.0.1:57581_solr) 
[n:127.0.0.1:57581_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1747757 INFO  
(zkCallback-3806-thread-2-processing-n:127.0.0.1:60333_solr) 
[n:127.0.0.1:60333_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1747757 INFO  
(zkCallback-3811-thread-1-processing-n:127.0.0.1:34551_solr) 
[n:127.0.0.1:34551_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1747757 INFO  
(zkCallback-3796-thread-1-processing-n:127.0.0.1:42420_solr) 
[n:127.0.0.1:42420_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1747758 INFO  
(zkCallback-3808-thread-2-processing-n:127.0.0.1:57581_solr) 
[n:127.0.0.1:57581_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1747759 INFO  
(zkCallback-3796-thread-2-processing-n:127.0.0.1:42420_solr) 
[n:127.0.0.1:42420_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1747758 INFO  
(zkCallback-3811-thread-2-processing-n:127.0.0.1:34551_solr) 
[n:127.0.0.1:34551_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1748647 INFO  (qtp840256473-17486) [n:127.0.0.1:34551_solr 
c:delDeadColl s:shard1 r:core_node3 x:delDeadColl_shard1_replica_n1] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1748653 INFO  (qtp2081593016-17467) [n:127.0.0.1:60333_solr 
c:delDeadColl s:shard2 r:core_node7 x:delDeadColl_shard2_replica_n4] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1748665 INFO  (qtp840256473-17486) [n:127.0.0.1:34551_solr 
c:delDeadColl s:shard1 r:core_node3 x:delDeadColl_shard1_replica_n1] 
o.a.s.s.IndexSchema [delDeadColl_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 1748667 INFO  (qtp840256473-17486) [n:127.0.0.1:34551_solr 
c:delDeadColl s:shard1 r:core_node3 x:delDeadColl_shard1_replica_n1] 
o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1748667 INFO  (qtp840256473-17486) [n:127.0.0.1:34551_solr 
c:delDeadColl s:shard1 r:core_node3 x:delDeadColl_shard1_replica_n1] 
o.a.s.c.CoreContainer Creating SolrCore 'delDeadColl_shard1_replica_n1' using 
configuration from collection delDeadColl, trusted=true
   [junit4]   2> 1748668 INFO  (qtp603739474-17477) [n:127.0.0.1:57581_solr 
c:delDeadColl s:shard2 r:core_node8 x:delDeadColl_shard2_replica_n6] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1748668 INFO  (qtp840256473-17486) [n:127.0.0.1:34551_solr 
c:delDeadColl s:shard1 r:core_node3 x:delDeadColl_shard1_replica_n1] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_34551.solr.core.delDeadColl.shard1.replica_n1' (registry 
'solr.core.delDeadColl.shard1.replica_n1') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@1c23a6f3
   [junit4]   2> 1748668 INFO  (qtp840256473-17486) [n:127.0.0.1:34551_solr 
c:delDeadColl s:shard1 r:core_node3 x:delDeadColl_shard1_replica_n1] 
o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1748668 INFO  (qtp840256473-17486) [n:127.0.0.1:34551_solr 
c:delDeadColl s:shard1 r:core_node3 x:delDeadColl_shard1_replica_n1] 
o.a.s.c.SolrCore [[delDeadColl_shard1_replica_n1] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteInactiveReplicaTest_8A256E9AC754A98D-001/tempDir-001/node4/delDeadColl_shard1_replica_n1],
 
dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteInactiveReplicaTest_8A256E9AC754A98D-001/tempDir-001/node4/./delDeadColl_shard1_replica_n1/data/]
   [junit4]   2> 1748670 INFO  (qtp2081593016-17467) [n:127.0.0.1:60333_solr 
c:delDeadColl s:shard2 r:core_node7 x:delDeadColl_shard2_replica_n4] 
o.a.s.s.IndexSchema [delDeadColl_shard2_replica_n4] Schema name=minimal
   [junit4]   2> 1748670 INFO  (qtp1934983229-17497) [n:127.0.0.1:42420_solr 
c:delDeadColl s:shard1 r:core_node5 x:delDeadColl_shard1_replica_n2] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1748672 INFO  (qtp2081593016-17467) [n:127.0.0.1:60333_solr 
c:delDeadColl s:shard2 r:core_node7 x:delDeadColl_shard2_replica_n4] 
o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1748672 INFO  (qtp2081593016-17467) [n:127.0.0.1:60333_solr 
c:delDeadColl s:shard2 r:core_node7 x:delDeadColl_shard2_replica_n4] 
o.a.s.c.CoreContainer Creating SolrCore 'delDeadColl_shard2_replica_n4' using 
configuration from collection delDeadColl, trusted=true
   [junit4]   2> 1748673 INFO  (qtp2081593016-17467) [n:127.0.0.1:60333_solr 
c:delDeadColl s:shard2 r:core_node7 x:delDeadColl_shard2_replica_n4] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_60333.solr.core.delDeadColl.shard2.replica_n4' (registry 
'solr.core.delDeadColl.shard2.replica_n4') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@1c23a6f3
   [junit4]   2> 1748673 INFO  (qtp2081593016-17467) [n:127.0.0.1:60333_solr 
c:delDeadColl s:shard2 r:core_node7 x:delDeadColl_shard2_replica_n4] 
o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1748673 INFO  (qtp2081593016-17467) [n:127.0.0.1:60333_solr 
c:delDeadColl s:shard2 r:core_node7 x:delDeadColl_shard2_replica_n4] 
o.a.s.c.SolrCore [[delDeadColl_shard2_replica_n4] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteInactiveReplicaTest_8A256E9AC754A98D-001/tempDir-001/node2/delDeadColl_shard2_replica_n4],
 
dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteInactiveReplicaTest_8A256E9AC754A98D-001/tempDir-001/node2/./delDeadColl_shard2_replica_n4/data/]
   [junit4]   2> 1748674 INFO  (qtp603739474-17477) [n:127.0.0.1:57581_solr 
c:delDeadColl s:shard2 r:core_node8 x:delDeadColl_shard2_replica_n6] 
o.a.s.s.IndexSchema [delDeadColl_shard2_replica_n6] Schema name=minimal
   [junit4]   2> 1748676 INFO  (qtp603739474-17477) [n:127.0.0.1:57581_solr 
c:delDeadColl s:shard2 r:core_node8 x:delDeadColl_shard2_replica_n6] 
o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1748676 INFO  (qtp603739474-17477) [n:127.0.0.1:57581_solr 
c:delDeadColl s:shard2 r:core_node8 x:delDeadColl_shard2_replica_n6] 
o.a.s.c.CoreContainer Creating SolrCore 'delDeadColl_shard2_replica_n6' using 
configuration from collection delDeadColl, trusted=true
   [junit4]   2> 1748676 INFO  (qtp1934983229-17497) [n:127.0.0.1:42420_solr 
c:delDeadColl s:shard1 r:core_node5 x:delDeadColl_shard1_replica_n2] 
o.a.s.s.IndexSchema [delDeadColl_shard1_replica_n2] Schema name=minimal
   [junit4]   2> 1748676 INFO  (qtp603739474-17477) [n:127.0.0.1:57581_solr 
c:delDeadColl s:shard2 r:core_node8 x:delDeadColl_shard2_replica_n6] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_57581.solr.core.delDeadColl.shard2.replica_n6' (registry 
'solr.core.delDeadColl.shard2.replica_n6') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@1c23a6f3
   [junit4]   2> 1748678 INFO  (qtp1934983229-17497) [n:127.0.0.1:42420_solr 
c:delDeadColl s:shard1 r:core_node5 x:delDeadColl_shard1_replica_n2] 
o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1748678 INFO  (qtp1934983229-17497) [n:127.0.0.1:42420_solr 
c:delDeadColl s:shard1 r:core_node5 x:delDeadColl_shard1_replica_n2] 
o.a.s.c.CoreContainer Creating SolrCore 'delDeadColl_shard1_replica_n2' using 
configuration from collection delDeadColl, trusted=true
   [junit4]   2> 1748678 INFO  (qtp1934983229-17497) [n:127.0.0.1:42420_solr 
c:delDeadColl s:shard1 r:core_node5 x:delDeadColl_shard1_replica_n2] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_42420.solr.core.delDeadColl.shard1.replica_n2' (registry 
'solr.core.delDeadColl.shard1.replica_n2') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@1c23a6f3
   [junit4]   2> 1748678 INFO  (qtp1934983229-17497) [n:127.0.0.1:42420_solr 
c:delDeadColl s:shard1 r:core_node5 x:delDeadColl_shard1_replica_n2] 
o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1748679 INFO  (qtp1934983229-17497) [n:127.0.0.1:42420_solr 
c:delDeadColl s:shard1 r:core_node5 x:delDeadColl_shard1_replica_n2] 
o.a.s.c.SolrCore [[delDeadColl_shard1_replica_n2] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteInactiveReplicaTest_8A256E9AC754A98D-001/tempDir-001/node1/delDeadColl_shard1_replica_n2],
 
dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteInactiveReplicaTest_8A256E9AC754A98D-001/tempDir-001/node1/./delDeadColl_shard1_replica_n2/data/]
   [junit4]   2> 1748682 INFO  (qtp603739474-17477) [n:127.0.0.1:57581_solr 
c:delDeadColl s:shard2 r:core_node8 x:delDeadColl_shard2_replica_n6] 
o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1748683 INFO  (qtp603739474-17477) [n:127.0.0.1:57581_solr 
c:delDeadColl s:shard2 r:core_node8 x:delDeadColl_shard2_replica_n6] 
o.a.s.c.SolrCore [[delDeadColl_shard2_replica_n6] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteInactiveReplicaTest_8A256E9AC754A98D-001/tempDir-001/node3/delDeadColl_shard2_replica_n6],
 
dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteInactiveReplicaTest_8A256E9AC754A98D-001/tempDir-001/node3/./delDeadColl_shard2_replica_n6/data/]
   [junit4]   2> 1748717 INFO  (qtp840256473-17486) [n:127.0.0.1:34551_solr 
c:delDeadColl s:shard1 r:core_node3 x:delDeadColl_shard1_replica_n1] 
o.a.s.u.UpdateHandler Using UpdateLog implementation: 
org.apache.solr.update.UpdateLog
   [junit4]   2> 1748717 INFO  (qtp840256473-17486) [n:127.0.0.1:34551_solr 
c:delDeadColl s:shard1 r:core_node3 x:delDeadColl_shard1_replica_n1] 
o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1748718 INFO  (qtp840256473-17486) [n:127.0.0.1:34551_solr 
c:delDeadColl s:shard1 r:core_node3 x:delDeadColl_shard1_replica_n1] 
o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1748718 INFO  (qtp840256473-17486) [n:127.0.0.1:34551_solr 
c:delDeadColl s:shard1 r:core_node3 x:delDeadColl_shard1_replica_n1] 
o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1748719 INFO  (qtp840256473-17486) [n:127.0.0.1:34551_solr 
c:delDeadColl s:shard1 r:core_node3 x:delDeadColl_shard1_replica_n1] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@3681013d[delDeadColl_shard1_replica_n1] main]
   [junit4]   2> 1748720 INFO  (qtp840256473-17486) [n:127.0.0.1:34551_solr 
c:delDeadColl s:shard1 r:core_node3 x:delDeadColl_shard1_replica_n1] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf
   [junit4]   2> 1748721 INFO  (qtp840256473-17486) [n:127.0.0.1:34551_solr 
c:delDeadColl s:shard1 r:core_node3 x:delDeadColl_shard1_replica_n1] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1748721 INFO  (qtp840256473-17486) [n:127.0.0.1:34551_solr 
c:delDeadColl s:shard1 r:core_node3 x:delDeadColl_shard1_replica_n1] 
o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1748723 INFO  (qtp1934983229-17497) [n:127.0.0.1:42420_solr 
c:delDeadColl s:shard1 r:core_node5 x:delDeadColl_shard1_replica_n2] 
o.a.s.u.UpdateHandler Using UpdateLog implementation: 
org.apache.solr.update.UpdateLog
   [junit4]   2> 1748723 INFO  (qtp1934983229-17497) [n:127.0.0.1:42420_solr 
c:delDeadColl s:shard1 r:core_node5 x:delDeadColl_shard1_replica_n2] 
o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1748724 INFO  (qtp1934983229-17497) [n:127.0.0.1:42420_solr 
c:delDeadColl s:shard1 r:core_node5 x:delDeadColl_shard1_replica_n2] 
o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1748724 INFO  (qtp1934983229-17497) [n:127.0.0.1:42420_solr 
c:delDeadColl s:shard1 r:core_node5 x:delDeadColl_shard1_replica_n2] 
o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1748740 INFO  (qtp840256473-17486) [n:127.0.0.1:34551_solr 
c:delDeadColl s:shard1 r:core_node3 x:delDeadColl_shard1_replica_n1] 
o.a.s.u.UpdateLog Could not find max version in index or recent updates, using 
new clock 1592415242979115008
   [junit4]   2> 1748743 INFO  (qtp1934983229-17497) [n:127.0.0.1:42420_solr 
c:delDeadColl s:shard1 r:core_node5 x:delDeadColl_shard1_replica_n2] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@2372958[delDeadColl_shard1_replica_n2] main]
   [junit4]   2> 1748744 INFO  (qtp1934983229-17497) [n:127.0.0.1:42420_solr 
c:delDeadColl s:shard1 r:core_node5 x:delDeadColl_shard1_replica_n2] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf
   [junit4]   2> 1748744 INFO  (qtp1934983229-17497) [n:127.0.0.1:42420_solr 
c:delDeadColl s:shard1 r:core_node5 x:delDeadColl_shard1_replica_n2] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1748745 INFO  
(searcherExecutor-8593-thread-1-processing-n:127.0.0.1:34551_solr 
x:delDeadColl_shard1_replica_n1 s:shard1 c:delDeadColl r:core_node3) 
[n:127.0.0.1:34551_solr c:delDeadColl s:shard1 r:core_node3 
x:delDeadColl_shard1_replica_n1] o.a.s.c.SolrCore 
[delDeadColl_shard1_replica_n1] Registered new searcher 
Searcher@3681013d[delDeadColl_shard1_replica_n1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1748745 INFO  (qtp1934983229-17497) [n:127.0.0.1:42420_solr 
c:delDeadColl s:shard1 r:core_node5 x:delDeadColl_shard1_replica_n2] 
o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1748745 INFO  (qtp1934983229-17497) [n:127.0.0.1:42420_solr 
c:delDeadColl s:shard1 r:core_node5 x:delDeadColl_shard1_replica_n2] 
o.a.s.u.UpdateLog Could not find max version in index or recent updates, using 
new clock 1592415242984357888
   [junit4]   2> 1748750 INFO  (qtp840256473-17486) [n:127.0.0.1:34551_solr 
c:delDeadColl s:shard1 r:core_node3 x:delDeadColl_shard1_replica_n1] 
o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for 
shard shard1: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 1748751 INFO  
(searcherExecutor-8596-thread-1-processing-n:127.0.0.1:42420_solr 
x:delDeadColl_shard1_replica_n2 s:shard1 c:delDeadColl r:core_node5) 
[n:127.0.0.1:42420_solr c:delDeadColl s:shard1 r:core_node5 
x:delDeadColl_shard1_replica_n2] o.a.s.c.SolrCore 
[delDeadColl_shard1_replica_n2] Registered new searcher 
Searcher@2372958[delDeadColl_shard1_replica_n2] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1748760 INFO  (qtp2081593016-17467) [n:127.0.0.1:60333_solr 
c:delDeadColl s:shard2 r:core_node7 x:delDeadColl_shard2_replica_n4] 
o.a.s.u.UpdateHandler Using UpdateLog implementation: 
org.apache.solr.update.UpdateLog
   [junit4]   2> 1748761 INFO  (qtp2081593016-17467) [n:127.0.0.1:60333_solr 
c:delDeadColl s:shard2 r:core_node7 x:delDeadColl_shard2_replica_n4] 
o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1748762 INFO  (qtp2081593016-17467) [n:127.0.0.1:60333_solr 
c:delDeadColl s:shard2 r:core_node7 x:delDeadColl_shard2_replica_n4] 
o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1748762 INFO  (qtp2081593016-17467) [n:127.0.0.1:60333_solr 
c:delDeadColl s:shard2 r:core_node7 x:delDeadColl_shard2_replica_n4] 
o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1748763 INFO  (qtp2081593016-17467) [n:127.0.0.1:60333_solr 
c:delDeadColl s:shard2 r:core_node7 x:delDeadColl_shard2_replica_n4] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@2cbd3a54[delDeadColl_shard2_replica_n4] main]
   [junit4]   2> 1748764 INFO  (qtp2081593016-17467) [n:127.0.0.1:60333_solr 
c:delDeadColl s:shard2 r:core_node7 x:delDeadColl_shard2_replica_n4] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf
   [junit4]   2> 1748764 INFO  (qtp2081593016-17467) [n:127.0.0.1:60333_solr 
c:delDeadColl s:shard2 r:core_node7 x:delDeadColl_shard2_replica_n4] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1748764 INFO  (qtp2081593016-17467) [n:127.0.0.1:60333_solr 
c:delDeadColl s:shard2 r:core_node7 x:delDeadColl_shard2_replica_n4] 
o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1748765 INFO  (qtp603739474-17477) [n:127.0.0.1:57581_solr 
c:delDeadColl s:shard2 r:core_node8 x:delDeadColl_shard2_replica_n6] 
o.a.s.u.UpdateHandler Using UpdateLog implementation: 
org.apache.solr.update.UpdateLog
   [junit4]   2> 1748765 INFO  (qtp603739474-17477) [n:127.0.0.1:57581_solr 
c:delDeadColl s:shard2 r:core_node8 x:delDeadColl_shard2_replica_n6] 
o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1748765 INFO  
(searcherExecutor-8594-thread-1-processing-n:127.0.0.1:60333_solr 
x:delDeadColl_shard2_replica_n4 s:shard2 c:delDeadColl r:core_node7) 
[n:127.0.0.1:60333_solr c:delDeadColl s:shard2 r:core_node7 
x:delDeadColl_shard2_replica_n4] o.a.s.c.SolrCore 
[delDeadColl_shard2_replica_n4] Registered new searcher 
Searcher@2cbd3a54[delDeadColl_shard2_replica_n4] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1748765 INFO  (qtp2081593016-17467) [n:127.0.0.1:60333_solr 
c:delDeadColl s:shard2 r:core_node7 x:delDeadColl_shard2_replica_n4] 
o.a.s.u.UpdateLog Could not find max version in index or recent updates, using 
new clock 1592415243005329408
   [junit4]   2> 1748766 INFO  (qtp603739474-17477) [n:127.0.0.1:57581_solr 
c:delDeadColl s:shard2 r:core_node8 x:delDeadColl_shard2_replica_n6] 
o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1748766 INFO  (qtp603739474-17477) [n:127.0.0.1:57581_solr 
c:delDeadColl s:shard2 r:core_node8 x:delDeadColl_shard2_replica_n6] 
o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1748767 INFO  (qtp603739474-17477) [n:127.0.0.1:57581_solr 
c:delDeadColl s:shard2 r:core_node8 x:delDeadColl_shard2_replica_n6] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@47de9db6[delDeadColl_shard2_replica_n6] main]
   [junit4]   2> 1748768 INFO  (qtp603739474-17477) [n:127.0.0.1:57581_solr 
c:delDeadColl s:shard2 r:core_node8 x:delDeadColl_shard2_replica_n6] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf
   [junit4]   2> 1748768 INFO  (qtp603739474-17477) [n:127.0.0.1:57581_solr 
c:delDeadColl s:shard2 r:core_node8 x:delDeadColl_shard2_replica_n6] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1748769 INFO  (qtp603739474-17477) [n:127.0.0.1:57581_solr 
c:delDeadColl s:shard2 r:core_node8 x:delDeadColl_shard2_replica_n6] 
o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1748769 INFO  
(searcherExecutor-8595-thread-1-processing-n:127.0.0.1:57581_solr 
x:delDeadColl_shard2_replica_n6 s:shard2 c:delDeadColl r:core_node8) 
[n:127.0.0.1:57581_solr c:delDeadColl s:shard2 r:core_node8 
x:delDeadColl_shard2_replica_n6] o.a.s.c.SolrCore 
[delDeadColl_shard2_replica_n6] Registered new searcher 
Searcher@47de9db6[delDeadColl_shard2_replica_n6] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1748769 INFO  (qtp603739474-17477) [n:127.0.0.1:57581_solr 
c:delDeadColl s:shard2 r:core_node8 x:delDeadColl_shard2_replica_n6] 
o.a.s.u.UpdateLog Could not find max version in index or recent updates, using 
new clock 1592415243009523712
   [junit4]   2> 1748772 INFO  (qtp2081593016-17467) [n:127.0.0.1:60333_solr 
c:delDeadColl s:shard2 r:core_node7 x:delDeadColl_shard2_replica_n4] 
o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for 
shard shard2: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 1748873 INFO  
(zkCallback-3808-thread-2-processing-n:127.0.0.1:57581_solr) 
[n:127.0.0.1:57581_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1748874 INFO  
(zkCallback-3808-thread-1-processing-n:127.0.0.1:57581_solr) 
[n:127.0.0.1:57581_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1748873 INFO  
(zkCallback-3806-thread-1-processing-n:127.0.0.1:60333_solr) 
[n:127.0.0.1:60333_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1748874 INFO  
(zkCallback-3796-thread-2-processing-n:127.0.0.1:42420_solr) 
[n:127.0.0.1:42420_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1748874 INFO  
(zkCallback-3806-thread-2-processing-n:127.0.0.1:60333_solr) 
[n:127.0.0.1:60333_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1748873 INFO  
(zkCallback-3811-thread-2-processing-n:127.0.0.1:34551_solr) 
[n:127.0.0.1:34551_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1748874 INFO  
(zkCallback-3796-thread-1-processing-n:127.0.0.1:42420_solr) 
[n:127.0.0.1:42420_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1748874 INFO  
(zkCallback-3811-thread-1-processing-n:127.0.0.1:34551_solr) 
[n:127.0.0.1:34551_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1749250 INFO  (qtp840256473-17486) [n:127.0.0.1:34551_solr 
c:delDeadColl s:shard1 r:core_node3 x:delDeadColl_shard1_replica_n1] 
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1749250 INFO  (qtp840256473-17486) [n:127.0.0.1:34551_solr 
c:delDeadColl s:shard1 r:core_node3 x:delDeadColl_shard1_replica_n1] 
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1749250 INFO  (qtp840256473-17486) [n:127.0.0.1:34551_solr 
c:delDeadColl s:shard1 r:core_node3 x:delDeadColl_shard1_replica_n1] 
o.a.s.c.SyncStrategy Sync replicas to 
https://127.0.0.1:34551/solr/delDeadColl_shard1_replica_n1/
   [junit4]   2> 1749251 INFO  (qtp840256473-17486) [n:127.0.0.1:34551_solr 
c:delDeadColl s:shard1 r:core_node3 x:delDeadColl_shard1_replica_n1] 
o.a.s.u.PeerSync PeerSync: core=delDeadColl_shard1_replica_n1 
url=https://127.0.0.1:34551/solr START 
replicas=[https://127.0.0.1:42420/solr/delDeadColl_shard1_replica_n2/] 
nUpdates=100
   [junit4]   2> 1749255 INFO  (qtp1934983229-17500) [n:127.0.0.1:42420_solr 
c:delDeadColl s:shard1 r:core_node5 x:delDeadColl_shard1_replica_n2] 
o.a.s.c.S.Request [delDeadColl_shard1_replica_n2]  webapp=/solr path=/get 
params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2}
 status=0 QTime=0
   [junit4]   2> 1749273 INFO  (qtp2081593016-17467) [n:127.0.0.1:60333_solr 
c:delDeadColl s:shard2 r:core_node7 x:delDeadColl_shard2_replica_n4] 
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1749273 INFO  (qtp2081593016-17467) [n:127.0.0.1:60333_solr 
c:delDeadColl s:shard2 r:core_node7 x:delDeadColl_shard2_replica_n4] 
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1749273 INFO  (qtp2081593016-17467) [n:127.0.0.1:60333_solr 
c:delDeadColl s:shard2 r:core_node7 x:delDeadColl_shard2_replica_n4] 
o.a.s.c.SyncStrategy Sync replicas to 
https://127.0.0.1:60333/solr/delDeadColl_shard2_replica_n4/
   [junit4]   2> 1749274 INFO  (qtp2081593016-17467) [n:127.0.0.1:60333_solr 
c:delDeadColl s:shard2 r:core_node7 x:delDeadColl_shard2_replica_n4] 
o.a.s.u.PeerSync PeerSync: core=delDeadColl_shard2_replica_n4 
url=https://127.0.0.1:60333/solr START 
replicas=[https://127.0.0.1:57581/solr/delDeadColl_shard2_replica_n6/] 
nUpdates=100
   [junit4]   2> 1749278 INFO  (qtp603739474-17479) [n:127.0.0.1:57581_solr 
c:delDeadColl s:shard2 r:core_node8 x:delDeadColl_shard2_replica_n6] 
o.a.s.c.S.Request [delDeadColl_shard2_replica_n6]  webapp=/solr path=/get 
params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2}
 status=0 QTime=0
   [junit4]   2> 1749552 INFO  (qtp840256473-17486) [n:127.0.0.1:34551_solr 
c:delDeadColl s:shard1 r:core_node3 x:delDeadColl_shard1_replica_n1] 
o.a.s.u.PeerSync PeerSync: core=delDeadColl_shard1_replica_n1 
url=https://127.0.0.1:34551/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 1749552 INFO  (qtp840256473-17486) [n:127.0.0.1:34551_solr 
c:delDeadColl s:shard1 r:core_node3 x:delDeadColl_shard1_replica_n1] 
o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the 
next candidate
   [junit4]   2> 1749552 INFO  (qtp840256473-17486) [n:127.0.0.1:34551_solr 
c:delDeadColl s:shard1 r:core_node3 x:delDeadColl_shard1_replica_n1] 
o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we 
can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 1749552 INFO  (qtp840256473-17486) [n:127.0.0.1:34551_solr 
c:delDeadColl s:shard1 r:core_node3 x:delDeadColl_shard1_replica_n1] 
o.a.s.c.ShardLeaderElectionContext Found all replicas participating in 
election, clear LIR
   [junit4]   2> 1749556 INFO  (qtp840256473-17486) [n:127.0.0.1:34551_solr 
c:delDeadColl s:shard1 r:core_node3 x:delDeadColl_shard1_replica_n1] 
o.a.s.c.ShardLeaderElectionContext I am the new leader: 
https://127.0.0.1:34551/solr/delDeadColl_shard1_replica_n1/ shard1
   [junit4]   2> 1749574 INFO  (qtp2081593016-17467) [n:127.0.0.1:60333_solr 
c:delDeadColl s:shard2 r:core_node7 x:delDeadColl_shard2_replica_n4] 
o.a.s.u.PeerSync PeerSync: core=delDeadColl_shard2_replica_n4 
url=https://127.0.0.1:60333/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 1749574 INFO  (qtp2081593016-17467) [n:127.0.0.1:60333_solr 
c:delDeadColl s:shard2 r:core_node7 x:delDeadColl_shard2_replica_n4] 
o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the 
next candidate
   [junit4]   2> 1749575 INFO  (qtp2081593016-17467) [n:127.0.0.1:60333_solr 
c:delDeadColl s:shard2 r:core_node7 x:delDeadColl_shard2_replica_n4] 
o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we 
can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 1749575 INFO  (qtp2081593016-17467) [n:127.0.0.1:60333_solr 
c:delDeadColl s:shard2 r:core_node7 x:delDeadColl_shard2_replica_n4] 
o.a.s.c.ShardLeaderElectionContext Found all replicas participating in 
election, clear LIR
   [junit4]   2> 1749578 INFO  (qtp2081593016-17467) [n:127.0.0.1:60333_solr 
c:delDeadColl s:shard2 r:core_node7 x:delDeadColl_shard2_replica_n4] 
o.a.s.c.ShardLeaderElectionContext I am the new leader: 
https://127.0.0.1:60333/solr/delDeadColl_shard2_replica_n4/ shard2
   [junit4]   2> 1749680 INFO  
(zkCallback-3808-thread-1-processing-n:127.0.0.1:57581_solr) 
[n:127.0.0.1:57581_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1749680 INFO  
(zkCallback-3796-thread-1-processing-n:127.0.0.1:42420_solr) 
[n:127.0.0.1:42420_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1749680 INFO  
(zkCallback-3811-thread-1-processing-n:127.0.0.1:34551_solr) 
[n:127.0.0.1:34551_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1749680 INFO  
(zkCallback-3806-thread-1-processing-n:127.0.0.1:60333_solr) 
[n:127.0.0.1:60333_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1749680 INFO  
(zkCallback-3806-thread-2-processing-n:127.0.0.1:60333_solr) 
[n:127.0.0.1:60333_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1749680 INFO  
(zkCallback-3811-thread-2-processing-n:127.0.0.1:34551_solr) 
[n:127.0.0.1:34551_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1749680 INFO  
(zkCallback-3808-thread-2-processing-n:127.0.0.1:57581_solr) 
[n:127.0.0.1:57581_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1749680 INFO  
(zkCallback-3796-thread-2-processing-n:127.0.0.1:42420_solr) 
[n:127.0.0.1:42420_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1749707 INFO  (qtp840256473-17486) [n:127.0.0.1:34551_solr 
c:delDeadColl s:shard1 r:core_node3 x:delDeadColl_shard1_replica_n1] 
o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1749710 INFO  (qtp840256473-17486) [n:127.0.0.1:34551_solr 
c:delDeadColl s:shard1 r:core_node3 x:delDeadColl_shard1_replica_n1] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=delDeadColl_shard1_replica_n1&action=CREATE&numShards=2&collection=delDeadColl&shard=shard1&wt=javabin&version=2&replicaType=NRT}
 status=0 QTime=2093
   [junit4]   2> 1749729 INFO  (qtp2081593016-17467) [n:127.0.0.1:60333_solr 
c:delDeadColl s:shard2 r:core_node7 x:delDeadColl_shard2_replica_n4] 
o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1749731 INFO  (qtp2081593016-17467) [n:127.0.0.1:60333_solr 
c:delDeadColl s:shard2 r:core_node7 x:delDeadColl_shard2_replica_n4] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&coreNodeName=core_node7&collection.configName=conf&newCollection=true&name=delDeadColl_shard2_replica_n4&action=CREATE&numShards=2&collection=delDeadColl&shard=shard2&wt=javabin&version=2&replicaType=NRT}
 status=0 QTime=2092
   [junit4]   2> 1749757 INFO  (qtp1934983229-17497) [n:127.0.0.1:42420_solr 
c:delDeadColl s:shard1 r:core_node5 x:delDeadColl_shard1_replica_n2] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf&newCollection=true&name=delDeadColl_shard1_replica_n2&action=CREATE&numShards=2&collection=delDeadColl&shard=shard1&wt=javabin&version=2&replicaType=NRT}
 status=0 QTime=2105
   [junit4]   2> 1749777 INFO  (qtp603739474-17477) [n:127.0.0.1:57581_solr 
c:delDeadColl s:shard2 r:core_node8 x:delDeadColl_shard2_replica_n6] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&coreNodeName=core_node8&collection.configName=conf&newCollection=true&name=delDeadColl_shard2_replica_n6&action=CREATE&numShards=2&collection=delDeadColl&shard=shard2&wt=javabin&version=2&replicaType=NRT}
 status=0 QTime=2127
   [junit4]   2> 1749780 INFO  (qtp840256473-17481) [n:127.0.0.1:34551_solr    
] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 
30 seconds. Check all shard replicas
   [junit4]   2> 1749877 INFO  
(zkCallback-3808-thread-1-processing-n:127.0.0.1:57581_solr) 
[n:127.0.0.1:57581_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1749877 INFO  
(zkCallback-3808-thread-2-processing-n:127.0.0.1:57581_solr) 
[n:127.0.0.1:57581_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1749877 INFO  
(zkCallback-3806-thread-1-processing-n:127.0.0.1:60333_solr) 
[n:127.0.0.1:60333_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1749877 INFO  
(zkCallback-3811-thread-2-processing-n:127.0.0.1:34551_solr) 
[n:127.0.0.1:34551_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1749877 INFO  
(zkCallback-3806-thread-2-processing-n:127.0.0.1:60333_solr) 
[n:127.0.0.1:60333_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1749877 INFO  
(zkCallback-3811-thread-1-processing-n:127.0.0.1:34551_solr) 
[n:127.0.0.1:34551_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1749877 INFO  
(zkCallback-3796-thread-2-processing-n:127.0.0.1:42420_solr) 
[n:127.0.0.1:42420_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1749877 INFO  
(zkCallback-3796-thread-1-processing-n:127.0.0.1:42420_solr) 
[n:127.0.0.1:42420_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1750780 INFO  (qtp840256473-17481) [n:127.0.0.1:34551_solr    
] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={replicationFactor=2&collection.configName=conf&maxShardsPerNode=2&name=delDeadColl&nrtReplicas=2&action=CREATE&numShards=2&wt=javabin&version=2}
 status=0 QTime=3491
   [junit4]   2> 1750783 INFO  
(TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[8A256E9AC754A98D])
 [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@32747ad9{SSL,[ssl, 
http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1750784 INFO  
(TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[8A256E9AC754A98D])
 [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1996784983
   [junit4]   2> 1750784 INFO  
(TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[8A256E9AC754A98D])
 [    ] o.a.s.m.SolrMetricManager Closing metric reporters for 
registry=solr.node, tag=null
   [junit4]   2> 1750784 INFO  
(TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[8A256E9AC754A98D])
 [    ] o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@21f5b3d2: rootName = 
solr_34551, domain = solr.node, service url = null, agent id = null] for 
registry solr.node / com.codahale.metrics.MetricRegistry@70f19010
   [junit4]   2> 1750790 INFO  
(TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[8A256E9AC754A98D])
 [    ] o.a.s.m.SolrMetricManager Closing metric reporters for 
registry=solr.jvm, tag=null
   [junit4]   2> 1750790 INFO  
(TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[8A256E9AC754A98D])
 [    ] o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@2467cbe1: rootName = 
solr_34551, domain = solr.jvm, service url = null, agent id = null] for 
registry solr.jvm / com.codahale.metrics.MetricRegistry@4e4e320
   [junit4]   2> 1750795 INFO  
(TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[8A256E9AC754A98D])
 [    ] o.a.s.m.SolrMetricManager Closing metric reporters for 
registry=solr.jetty, tag=null
   [junit4]   2> 1750795 INFO  
(TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[8A256E9AC754A98D])
 [    ] o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@23b9ac8d: rootName = 
solr_34551, domain = solr.jetty, service url = null, agent id = null] for 
registry solr.jetty / com.codahale.metrics.MetricRegistry@db6764
   [junit4]   2> 1750796 INFO  
(TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[8A256E9AC754A98D])
 [    ] o.a.s.c.ZkController Remove node as live in 
ZooKeeper:/live_nodes/127.0.0.1:34551_solr
   [junit4]   2> 1750797 INFO  (zkCallback-3824-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 1750797 INFO  
(zkCallback-3808-thread-2-processing-n:127.0.0.1:57581_solr) 
[n:127.0.0.1:57581_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (4) -> (3)
   [junit4]   2> 1750798 INFO  
(zkCallback-3806-thread-1-processing-n:127.0.0.1:60333_solr) 
[n:127.0.0.1:60333_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (4) -> (3)
   [junit4]   2> 1750798 INFO  
(TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[8A256E9AC754A98D])
 [    ] o.a.s.m.SolrMetricManager Closing metric reporters for 
registry=solr.cluster, tag=null
   [junit4]   2> 1750797 INFO  
(zkCallback-3796-thread-3-processing-n:127.0.0.1:42420_solr) 
[n:127.0.0.1:42420_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (4) -> (3)
   [junit4]   2> 1750798 INFO  
(zkCallback-3811-thread-2-processing-n:127.0.0.1:34551_solr) 
[n:127.0.0.1:34551_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (4) -> (3)
   [junit4]   2> 1750799 INFO  (coreCloseExecutor-8613-thread-1) 
[n:127.0.0.1:34551_solr c:delDeadColl s:shard1 r:core_node3 
x:delDeadColl_shard1_replica_n1] o.a.s.c.SolrCore 
[delDeadColl_shard1_replica_n1]  CLOSING SolrCore 
org.apache.solr.core.SolrCore@b08cb3f
   [junit4]   2> 1750799 INFO  (coreCloseExecutor-8613-thread-1) 
[n:127.0.0.1:34551_solr c:delDeadColl s:shard1 r:core_node3 
x:delDeadColl_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric 
reporters for registry=solr.core.delDeadColl.shard1.replica_n1, tag=185125695
   [junit4]   2> 1750799 INFO  (coreCloseExecutor-8613-thread-1) 
[n:127.0.0.1:34551_solr c:delDeadColl s:shard1 r:core_node3 
x:delDeadColl_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@726542a7: rootName = 
solr_34551, domain = solr.core.delDeadColl.shard1.replica_n1, service url = 
null, agent id = null] for registry solr.core.delDeadColl.shard1.replica_n1 / 
com.codahale.metrics.MetricRegistry@34002f5
   [junit4]   2> 1750811 INFO  (coreCloseExecutor-8613-thread-1) 
[n:127.0.0.1:34551_solr c:delDeadColl s:shard1 r:core_node3 
x:delDeadColl_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric 
reporters for registry=solr.collection.delDeadColl.shard1.leader, tag=185125695
   [junit4]   2> 1750817 INFO  
(zkCallback-3796-thread-3-processing-n:127.0.0.1:42420_solr) 
[n:127.0.0.1:42420_solr c:delDeadColl s:shard1 r:core_node5 
x:delDeadColl_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I may be 
the new leader - try and sync
   [junit4]   2> 1750818 INFO  
(TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[8A256E9AC754A98D])
 [    ] o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@49d138ba{/solr,null,UNAVAILABLE}
   [junit4]   2> 1750818 INFO  
(TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[8A256E9AC754A98D])
 [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 1750819 INFO  
(TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[8A256E9AC754A98D])
 [    ] o.a.s.c.DeleteInactiveReplicaTest Removing replica shard1/core_node3 
   [junit4]   2> 1750821 INFO  (qtp1934983229-17499) [n:127.0.0.1:42420_solr    
] o.a.s.h.a.CollectionsHandler Invoked Collection Action :deletereplica with 
params 
replica=core_node3&action=DELETEREPLICA&collection=delDeadColl&shard=shard1&wt=javabin&version=2
 and sendToOCPQueue=true
   [junit4]   2> 1750823 INFO  
(OverseerCollectionConfigSetProcessor-73296055638622215-127.0.0.1:42420_solr-n_0000000000)
 [n:127.0.0.1:42420_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: 
/overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may 
have disconnected from ZooKeeper
   [junit4]   2> 1750924 INFO  
(zkCallback-3806-thread-1-processing-n:127.0.0.1:60333_solr) 
[n:127.0.0.1:60333_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [3])
   [junit4]   2> 1750924 INFO  
(zkCallback-3808-thread-1-processing-n:127.0.0.1:57581_solr) 
[n:127.0.0.1:57581_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [3])
   [junit4]   2> 1750924 INFO  
(zkCallback-3808-thread-2-processing-n:127.0.0.1:57581_solr) 
[n:127.0.0.1:57581_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [3])
   [junit4]   2> 1750924 INFO  
(zkCallback-3796-thread-1-processing-n:127.0.0.1:42420_solr) 
[n:127.0.0.1:42420_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [3])
   [junit4]   2> 1750924 INFO  
(zkCallback-3796-thread-2-processing-n:127.0.0.1:42420_solr) 
[n:127.0.0.1:42420_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [3])
   [junit4]   2> 1750924 INFO  
(zkCallback-3806-thread-2-processing-n:127.0.0.1:60333_solr) 
[n:127.0.0.1:60333_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [3])
   [junit4]   2> 1751025 INFO  (qtp1934983229-17499) [n:127.0.0.1:42420_solr    
] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={replica=core_node3&action=DELETEREPLICA&collection=delDeadColl&shard=shard1&wt=javabin&version=2}
 status=0 QTime=204
   [junit4]   2> 1751033 INFO  
(TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[8A256E9AC754A98D])
 [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 
2017-11-21T16:27:37-05:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1751035 INFO  
(TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[8A256E9AC754A98D])
 [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1751035 INFO  
(TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[8A256E9AC754A98D])
 [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1751035 INFO  
(TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[8A256E9AC754A98D])
 [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 1751035 INFO  
(TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[8A256E9AC754A98D])
 [    ] o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@1868fd7f{/solr,null,AVAILABLE}
   [junit4]   2> 1751036 INFO  
(TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[8A256E9AC754A98D])
 [    ] o.e.j.s.AbstractConnector Started ServerConnector@1bfcde19{SSL,[ssl, 
http/1.1]}{127.0.0.1:49038}
   [junit4]   2> 1751036 INFO  
(TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[8A256E9AC754A98D])
 [    ] o.e.j.s.Server Started @1753315ms
   [junit4]   2> 1751036 INFO  
(TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[8A256E9AC754A98D])
 [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=49038}
   [junit4]   2> 1751036 ERROR 
(TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[8A256E9AC754A98D])
 [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging 
may be missing or incomplete.
   [junit4]   2> 1751037 INFO  
(TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[8A256E9AC754A98D])
 [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ 
version 8.0.0
   [junit4]   2> 1751037 INFO  
(TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[8A256E9AC754A98D])
 [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on 
port null
   [junit4]   2> 1751037 INFO  
(TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[8A256E9AC754A98D])
 [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1751037 INFO  
(TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[8A256E9AC754A98D])
 [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2018-02-14T21:58:40.580Z
   [junit4]   2> 1751038 INFO  (zkConnectionManagerCallback-3827-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1751039 INFO  
(TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[8A256E9AC754A98D])
 [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1751048 INFO  
(TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[8A256E9AC754A98D])
 [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34083/solr
   [junit4]   2> 1751051 INFO  (zkConnectionManagerCallback-3831-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1751052 INFO  
(zkConnectionManagerCallback-3833-thread-1-processing-n:127.0.0.1:49038_solr) 
[n:127.0.0.1:49038_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1751056 INFO  
(TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[8A256E9AC754A98D])
 [n:127.0.0.1:49038_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (3)
   [junit4]   2> 1751057 INFO  
(TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[8A256E9AC754A98D])
 [n:127.0.0.1:49038_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1751058 INFO  
(TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[8A256E9AC754A98D])
 [n:127.0.0.1:49038_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:49038_solr
   [junit4]   2> 1751059 INFO  (zkCallback-3824-thread-2) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1751059 INFO  
(zkCallback-3806-thread-2-processing-n:127.0.0.1:60333_solr) 
[n:127.0.0.1:60333_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (3) -> (4)
   [junit4]   2> 1751059 INFO  
(zkCallback-3796-thread-1-processing-n:127.0.0.1:42420_solr) 
[n:127.0.0.1:42420_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (3) -> (4)
   [junit4]   2> 1751060 INFO  
(zkCallback-3808-thread-2-processing-n:127.0.0.1:57581_solr) 
[n:127.0.0.1:57581_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (3) -> (4)
   [junit4]   2> 1751060 INFO  
(zkCallback-3832-thread-1-processing-n:127.0.0.1:49038_solr) 
[n:127.0.0.1:49038_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (3) -> (4)
   [junit4]   2> 1751150 INFO  
(TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[8A256E9AC754A98D])
 [n:127.0.0.1:49038_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_49038.solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@1c23a6f3
   [junit4]   2> 1751159 INFO  
(TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[8A256E9AC754A98D])
 [n:127.0.0.1:49038_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_49038.solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@1c23a6f3
   [junit4]   2> 1751159 INFO  
(TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[8A256E9AC754A98D])
 [n:127.0.0.1:49038_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_49038.solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@1c23a6f3
   [junit4]   2> 1751161 INFO  
(TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[8A256E9AC754A98D])
 [n:127.0.0.1:49038_solr    ] o.a.s.c.CorePropertiesLocator Found 1 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteInactiveReplicaTest_8A256E9AC754A98D-001/tempDir-001/node4/.
   [junit4]   2> 1751161 INFO  
(TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[8A256E9AC754A98D])
 [n:127.0.0.1:49038_solr    ] o.a.s.c.CorePropertiesLocator Cores are: 
[delDeadColl_shard1_replica_n1]
   [junit4]   2> 1751163 INFO  
(coreLoadExecutor-8620-thread-1-processing-n:127.0.0.1:49038_solr) 
[n:127.0.0.1:49038_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating 
transient cache for 2147483647 transient cores
   [junit4]   2> 1751171 INFO  
(coreLoadExecutor-8620-thread-1-processing-n:127.0.0.1:49038_solr) 
[n:127.0.0.1:49038_solr    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1751177 INFO  
(coreLoadExecutor-8620-thread-1-processing-n:127.0.0.1:49038_solr) 
[n:127.0.0.1:49038_solr    ] o.a.s.s.IndexSchema 
[delDeadColl_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 1751179 INFO  
(coreLoadExecutor-8620-thread-1-processing-n:127.0.0.1:49038_solr) 
[n:127.0.0.1:49038_solr    ] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with 
uniqueid field id
   [junit4]   2> 1752824 INFO  
(OverseerCollectionConfigSetProcessor-73296055638622215-127.0.0.1:42420_solr-n_0000000000)
 [n:127.0.0.1:42420_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: 
/overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may 
have disconnected from ZooKeeper
   [junit4]   2> 1753317 INFO  
(zkCallback-3796-thread-3-processing-n:127.0.0.1:42420_solr) 
[n:127.0.0.1:42420_solr c:delDeadColl s:shard1 r:core_node5 
x:delDeadColl_shard1_replica_n2] o.a.s.c.SyncStrategy Sync replicas to 
https://127.0.0.1:42420/solr/delDeadColl_shard1_replica_n2/
   [junit4]   2> 1753318 INFO  
(zkCallback-3796-thread-3-processing-n:127.0.0.1:42420_solr) 
[n:127.0.0.1:42420_solr c:delDeadColl s:shard1 r:core_node5 
x:delDeadColl_shard1_replica_n2] o.a.s.c.SyncStrategy Sync Success - now sync 
replicas to me
   [junit4]   2> 1753318 INFO  
(zkCallback-3796-thread-3-processing-n:127.0.0.1:42420_solr) 
[n:127.0.0.1:42420_solr c:delDeadColl s:shard1 r:core_node5 
x:delDeadColl_shard1_replica_n2] o.a.s.c.SyncStrategy 
https://127.0.0.1:42420/solr/delDeadColl_shard1_replica_n2/ has no replicas
   [junit4]   2> 1753320 INFO  
(zkCallback-3796-thread-3-processing-n:127.0.0.1:42420_solr) 
[n:127.0.0.1:42420_solr c:delDeadColl s:shard1 r:core_node5 
x:delDeadColl_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I am the 
new leader: https://127.0.0.1:42420/solr/delDeadColl_shard1_replica_n2/ shard1
   [junit4]   2> 1753321 INFO  
(zkCallback-3796-thread-1-processing-n:127.0.0.1:42420_solr) 
[n:127.0.0.1:42420_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1753321 INFO  
(zkCallback-3796-thread-3-processing-n:127.0.0.1:42420_solr) 
[n:127.0.0.1:42420_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1753321 INFO  
(zkCallback-3832-thread-1-processing-n:127.0.0.1:49038_solr) 
[n:127.0.0.1:49038_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1753321 INFO  
(zkCallback-3806-thread-2-processing-n:127.0.0.1:60333_solr) 
[n:127.0.0.1:60333_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1753322 INFO  
(zkCallback-3806-thread-1-processing-n:127.0.0.1:60333_solr) 
[n:127.0.0.1:60333_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1753322 INFO  
(zkCallback-3808-thread-2-processing-n:127.0.0.1:57581_solr) 
[n:127.0.0.1:57581_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1753322 INFO  
(zkCallback-3808-thread-1-processing-n:127.0.0.1:57581_solr) 
[n:127.0.0.1:57581_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/delDeadColl/state.json] for collection [delDeadColl] has 
occurred - updating... (live nodes size: [4])
   [junit4]   2> 1754180 ERROR 
(coreLoadExecutor-8620-thread-1-processing-n:127.0.0.1:49038_solr) 
[n:127.0.0.1:49038_solr c:delDeadColl s:shard1 r:core_node3 
x:delDeadColl_shard1_replica_n1] o.a.s.c.ZkController 
   [junit4]   2> org.apache.solr.common.SolrException: coreNodeName core_node3 
does not exist in shard shard1: 
DocCollection(delDeadColl//collections/delDeadColl/state.json/7)={
   [junit4]   2>   "pullReplicas":"0",
   [junit4]   2>   "replicationFactor":"2",
   [junit4]   2>   "shards":{
   [junit4]   2>     "shard1":{
   [junit4]   2>       "range":"80000000-ffffffff",
   [junit4]   2>       "state":"active",
   [junit4]   2>       "replicas":{"core_node5":{
   [junit4]   2>           "core":"delDeadColl_shard1_replica_n2",
   [junit4]   2>           "base_url":"https://127.0.0.1:42420/solr";,
   [junit4]   2>           "node_name":"127.0.0.1:42420_solr",
   [junit4]   2>           "state":"active",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "leader":"tr

[...truncated too long message...]

kSupport.parkNanos(LockSupport.java:215)
   [junit4]   2>         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
   [junit4]   2>         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
   [junit4]   2>         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
   [junit4]   2>         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
   [junit4]   2>         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: test params are: 
codec=FastCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST,
 chunkSize=3315, maxDocsPerChunk=166, blockSize=771), 
termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST, 
chunkSize=3315, blockSize=771)), 
sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@33953dd7),
 locale=sl, timezone=SystemV/MST7
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 
1.8.0_144 (64-bit)/cpus=4,threads=2,free=148871384,total=524812288
   [junit4]   2> NOTE: All tests run in this JVM: [TestAnalyzeInfixSuggestions, 
TestPolicyCloud, BlockJoinFacetDistribTest, MoveReplicaTest, MetricsConfigTest, 
HighlighterTest, CdcrRequestHandlerTest, TestBinaryResponseWriter, 
InfoHandlerTest, TestSolr4Spatial, SolrMetricReporterTest, SSLMigrationTest, 
LeaderElectionContextKeyTest, BasicZkTest, 
LeaderInitiatedRecoveryOnShardRestartTest, TestDownShardTolerantSearch, 
TestSolrXml, TestFieldCacheVsDocValues, CollectionReloadTest, TestCryptoKeys, 
PolyFieldTest, TestFunctionQuery, TestFieldCacheReopen, 
DistributedFacetPivotLongTailTest, TestExactSharedStatsCache, 
HdfsLockFactoryTest, SolrCLIZkUtilsTest, TestStressVersions, 
TestStandardQParsers, MergeStrategyTest, TestNumericTokenStream, 
TermsComponentTest, TestSolrJ, QueryEqualityTest, TestComputePlanAction, 
TestFoldingMultitermQuery, ManagedSchemaRoundRobinCloudTest, FileUtilsTest, 
TestConfigSetsAPI, TestSerializedLuceneMatchVersion, TestStressRecovery, 
TestSearcherReuse, TestEmbeddedSolrServerAdminHandler, BooleanFieldTest, 
ZkControllerTest, TestApiFramework, TestLegacyFieldReuse, ZkFailoverTest, 
TestDynamicFieldResource, TestMissingGroups, DistanceUnitsTest, 
TestCoreAdminApis, CoreSorterTest, DirectSolrConnectionTest, 
TestLocalFSCloudBackupRestore, OutOfBoxZkACLAndCredentialsProvidersTest, 
HdfsBasicDistributedZk2Test, DistributedSuggestComponentTest, 
TestObjectReleaseTracker, ShardRoutingCustomTest, DOMUtilTest, 
TestShardHandlerFactory, TestZkChroot, TestRestoreCore, SyncSliceTest, 
TestAddFieldRealTimeGet, TestCharFilters, CurrencyRangeFacetCloudTest, 
SOLR749Test, TestSha256AuthenticationProvider, SimplePostToolTest, 
OverseerTest, PrimUtilsTest, SuggesterFSTTest, HdfsChaosMonkeySafeLeaderTest, 
SliceStateTest, TestSolr4Spatial2, RequestLoggingTest, TestConfigSets, 
SolrInfoBeanTest, SuggestComponentContextFilterQueryTest, 
UpdateRequestProcessorFactoryTest, SignatureUpdateProcessorFactoryTest, 
TestRTGBase, TestSweetSpotSimilarityFactory, PrimitiveFieldTypeTest, 
DistributedIntervalFacetingTest, BasicFunctionalityTest, TestSolrIndexConfig, 
TestFilteredDocIdSet, DeleteStatusTest, TestPostingsSolrHighlighter, 
DocumentBuilderTest, TestMacros, DeleteNodeTest, DistribJoinFromCollectionTest, 
TestCollectionAPI, TestGenericDistributedQueue, TestElisionMultitermQuery, 
CollectionStateFormat2Test, TestBM25SimilarityFactory, TestXmlQParser, 
GraphQueryTest, HdfsCollectionsAPIDistributedZkTest, TestConfigOverlay, 
AsyncCallRequestStatusResponseTest, CoreAdminCreateDiscoverTest, HLLUtilTest, 
TestCloudJSONFacetJoinDomain, AddReplicaTest, CollectionsAPIDistributedZkTest, 
TestOverriddenPrefixQueryForCustomFieldType, TestTrackingShardHandlerFactory, 
TestFreeTextSuggestions, TestGeoJSONResponseWriter, TestHdfsBackupRestoreCore, 
TestTestInjection, PeerSyncTest, BadComponentTest, 
TestFuzzyAnalyzedSuggestions, AutoAddReplicasIntegrationTest, 
TestReplicationHandler, TestSystemIdResolver, QueryElevationComponentTest, 
TestSchemalessBufferedUpdates, TestBlobHandler, TestJsonFacetRefinement, 
TestMultiValuedNumericRangeQuery, ChaosMonkeySafeLeaderTest, 
BasicDistributedZk2Test, UnloadDistributedZkTest, OpenCloseCoreStressTest, 
TestDistributedSearch, TestDistributedGrouping, TestFaceting, TestRecovery, 
DistributedSpellCheckComponentTest, TestJoin, TestReload, 
DistributedTermsComponentTest, SimpleFacetsTest, SolrCoreTest, 
SpellCheckComponentTest, TestGroupingSearch, SolrCmdDistributorTest, TestSort, 
DirectUpdateHandlerTest, TestBadConfig, SoftAutoCommitTest, 
ShowFileRequestHandlerTest, SolrIndexSplitterTest, AnalysisAfterCoreReloadTest, 
TestExtendedDismaxParser, CoreAdminHandlerTest, SuggesterTest, TestTrie, 
TestCSVLoader, NoCacheHeaderTest, SchemaVersionSpecificBehaviorTest, 
TestUpdate, FieldMutatingUpdateProcessorTest, TestAtomicUpdateErrorCases, 
DirectUpdateHandlerOptimizeTest, DefaultValueUpdateProcessorTest, 
SortByFunctionTest, TestRemoteStreaming, DocValuesMultiTest, 
DistanceFunctionTest, XsltUpdateRequestHandlerTest, DebugComponentTest, 
TestSurroundQueryParser, TestQueryUtils, DirectSolrSpellCheckerTest, 
TestReversedWildcardFilterFactory, TestQueryTypes, TestOmitPositions, 
XmlUpdateRequestHandlerTest, PathHierarchyTokenizerFactoryTest, 
TermVectorComponentTest, TestIndexingPerformance, MoreLikeThisHandlerTest, 
RequiredFieldsTest, FieldAnalysisRequestHandlerTest, FastVectorHighlighterTest, 
LoggingHandlerTest, RegexBoostProcessorTest, IndexSchemaTest, 
TestJmxIntegration, TestCollationField, QueryParsingTest, 
TestAnalyzedSuggestions, JsonLoaderTest, TestPartialUpdateDeduplication, 
TestPHPSerializedResponseWriter, PingRequestHandlerTest, CSVRequestHandlerTest, 
TestComponentsName, HighlighterConfigTest, AlternateDirectoryTest, 
TestQuerySenderNoQuery, CopyFieldTest, SolrIndexConfigTest, 
TestMergePolicyConfig, TestSolrDeletionPolicy2, MultiTermTest, SampleTest, 
TestDocSet, TestBinaryField, TestSearchPerf, TestConfig, OutputWriterTest, 
TestLuceneMatchVersion, TestPhraseSuggestions, TestCodecSupport, 
TestXIncludeConfig, TestDFRSimilarityFactory, TestPerFieldSimilarity, 
TestLMDirichletSimilarityFactory, TestLMJelinekMercerSimilarityFactory, 
TimeZoneUtilsTest, ResourceLoaderTest, ScriptEngineTest, TestFastLRUCache, 
ChaosMonkeyNothingIsSafeTest, PreAnalyzedFieldTest, 
TestSuggestSpellingConverter, DateFieldTest, RAMDirectoryFactoryTest, 
TestLRUCache, TestUtils, TestDocumentBuilder, SystemInfoHandlerTest, 
CircularListTest, DistributedMLTComponentTest, SolrTestCaseJ4Test, 
TestCursorMarkWithoutUniqueKey, TestDistributedMissingSort, 
TestHighlightDedupGrouping, TestSimpleTrackingShardHandler, 
TestLuceneIndexBackCompat, TestJettySolrRunner, ConnectionReuseTest, 
ChaosMonkeyNothingIsSafeWithPullReplicasTest, CloudExitableDirectoryReaderTest, 
CollectionsAPISolrJTest, ConcurrentCreateRoutedAliasTest, ConfigSetsAPITest, 
ConnectionManagerTest, CreateRoutedAliasTest, DeleteInactiveReplicaTest]
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=DeleteInactiveReplicaTest -Dtests.seed=8A256E9AC754A98D 
-Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=sl 
-Dtests.timezone=SystemV/MST7 -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J1 | DeleteInactiveReplicaTest (suite) <<<
   [junit4]    > Throwable #1: 
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE 
scope at org.apache.solr.cloud.DeleteInactiveReplicaTest: 
   [junit4]    >    1) Thread[id=17650, name=qtp2081593016-17650, 
state=TIMED_WAITING, group=TGRP-DeleteInactiveReplicaTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([8A256E9AC754A98D]:0)Throwable #2: 
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie 
threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=17650, name=qtp2081593016-17650, 
state=TIMED_WAITING, group=TGRP-DeleteInactiveReplicaTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([8A256E9AC754A98D]:0)
   [junit4] Completed [493/776 (1!)] on J1 in 60.89s, 1 test, 2 errors <<< 
FAILURES!

[...truncated 49382 lines...]
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Reply via email to