Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/17930/
Java: 32bit/jdk-9-ea+136 -server -XX:+UseParallelGC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteNodeTest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.DeleteNodeTest:     
1) Thread[id=3972, 
name=OverseerHdfsCoreFailoverThread-96676649508012050-127.0.0.1:37968_solr-n_0000000001,
 state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]         at 
java.lang.Thread.sleep(java.base@9-ea/Native Method)         at 
org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)
         at java.lang.Thread.run(java.base@9-ea/Thread.java:843)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE 
scope at org.apache.solr.cloud.DeleteNodeTest: 
   1) Thread[id=3972, 
name=OverseerHdfsCoreFailoverThread-96676649508012050-127.0.0.1:37968_solr-n_0000000001,
 state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
        at java.lang.Thread.sleep(java.base@9-ea/Native Method)
        at 
org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)
        at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
        at __randomizedtesting.SeedInfo.seed([8313F26FCB3FBD32]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteNodeTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) 
Thread[id=3972, 
name=OverseerHdfsCoreFailoverThread-96676649508012050-127.0.0.1:37968_solr-n_0000000001,
 state=RUNNABLE, group=Overseer Hdfs SolrCore Failover Thread.]         at 
java.lang.Throwable.fillInStackTrace(java.base@9-ea/Native Method)         at 
java.lang.Throwable.fillInStackTrace(java.base@9-ea/Throwable.java:788)         
at java.lang.Throwable.<init>(java.base@9-ea/Throwable.java:271)         at 
java.lang.Exception.<init>(java.base@9-ea/Exception.java:66)         at 
java.lang.InterruptedException.<init>(java.base@9-ea/InterruptedException.java:67)
         at java.lang.Thread.sleep(java.base@9-ea/Native Method)         at 
org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)
         at java.lang.Thread.run(java.base@9-ea/Thread.java:843)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie 
threads that couldn't be terminated:
   1) Thread[id=3972, 
name=OverseerHdfsCoreFailoverThread-96676649508012050-127.0.0.1:37968_solr-n_0000000001,
 state=RUNNABLE, group=Overseer Hdfs SolrCore Failover Thread.]
        at java.lang.Throwable.fillInStackTrace(java.base@9-ea/Native Method)
        at 
java.lang.Throwable.fillInStackTrace(java.base@9-ea/Throwable.java:788)
        at java.lang.Throwable.<init>(java.base@9-ea/Throwable.java:271)
        at java.lang.Exception.<init>(java.base@9-ea/Exception.java:66)
        at 
java.lang.InterruptedException.<init>(java.base@9-ea/InterruptedException.java:67)
        at java.lang.Thread.sleep(java.base@9-ea/Native Method)
        at 
org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)
        at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
        at __randomizedtesting.SeedInfo.seed([8313F26FCB3FBD32]:0)




Build Log:
[...truncated 11072 lines...]
   [junit4] Suite: org.apache.solr.cloud.DeleteNodeTest
   [junit4]   2> Creating dataDir: 
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteNodeTest_8313F26FCB3FBD32-001/init-core-data-001
   [junit4]   2> 554228 INFO  
(SUITE-DeleteNodeTest-seed#[8313F26FCB3FBD32]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: 
@org.apache.solr.util.RandomizeSSL(reason="", ssl=0.0/0.0, value=0.0/0.0, 
clientAuth=0.0/0.0)
   [junit4]   2> 554228 INFO  
(SUITE-DeleteNodeTest-seed#[8313F26FCB3FBD32]-worker) [    ] 
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 554228 INFO  (Thread-719) [    ] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 554228 INFO  (Thread-719) [    ] o.a.s.c.ZkTestServer Starting 
server
   [junit4]   2> 554328 INFO  
(SUITE-DeleteNodeTest-seed#[8313F26FCB3FBD32]-worker) [    ] 
o.a.s.c.ZkTestServer start zk server on port:34578
   [junit4]   2> 554333 INFO  (jetty-launcher-749-thread-2) [    ] 
o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 554333 INFO  (jetty-launcher-749-thread-1) [    ] 
o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 554333 INFO  (jetty-launcher-749-thread-4) [    ] 
o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 554333 INFO  (jetty-launcher-749-thread-3) [    ] 
o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 554334 INFO  (jetty-launcher-749-thread-5) [    ] 
o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 554338 INFO  (jetty-launcher-749-thread-6) [    ] 
o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 554338 INFO  (jetty-launcher-749-thread-2) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@14be3e1{/solr,null,AVAILABLE}
   [junit4]   2> 554339 INFO  (jetty-launcher-749-thread-4) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@7abb53{/solr,null,AVAILABLE}
   [junit4]   2> 554339 INFO  (jetty-launcher-749-thread-1) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@1b44cb9{/solr,null,AVAILABLE}
   [junit4]   2> 554340 INFO  (jetty-launcher-749-thread-2) [    ] 
o.e.j.s.ServerConnector Started 
ServerConnector@1bf8201{HTTP/1.1,[http/1.1]}{127.0.0.1:32944}
   [junit4]   2> 554340 INFO  (jetty-launcher-749-thread-4) [    ] 
o.e.j.s.ServerConnector Started 
ServerConnector@befbd3{HTTP/1.1,[http/1.1]}{127.0.0.1:37968}
   [junit4]   2> 554340 INFO  (jetty-launcher-749-thread-1) [    ] 
o.e.j.s.ServerConnector Started 
ServerConnector@dbd6d9{HTTP/1.1,[http/1.1]}{127.0.0.1:42537}
   [junit4]   2> 554340 INFO  (jetty-launcher-749-thread-4) [    ] 
o.e.j.s.Server Started @556809ms
   [junit4]   2> 554340 INFO  (jetty-launcher-749-thread-1) [    ] 
o.e.j.s.Server Started @556810ms
   [junit4]   2> 554342 INFO  (jetty-launcher-749-thread-4) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=37968}
   [junit4]   2> 554340 INFO  (jetty-launcher-749-thread-2) [    ] 
o.e.j.s.Server Started @556809ms
   [junit4]   2> 554342 INFO  (jetty-launcher-749-thread-1) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=42537}
   [junit4]   2> 554342 INFO  (jetty-launcher-749-thread-2) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=32944}
   [junit4]   2> 554342 INFO  (jetty-launcher-749-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 
7.0.0
   [junit4]   2> 554342 INFO  (jetty-launcher-749-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 
7.0.0
   [junit4]   2> 554342 INFO  (jetty-launcher-749-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 554342 INFO  (jetty-launcher-749-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 
7.0.0
   [junit4]   2> 554342 INFO  (jetty-launcher-749-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 554342 INFO  (jetty-launcher-749-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 554342 INFO  (jetty-launcher-749-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 554343 INFO  (jetty-launcher-749-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 554343 INFO  (jetty-launcher-749-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2016-09-29T17:03:22.226289Z
   [junit4]   2> 554343 INFO  (jetty-launcher-749-thread-3) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@15d261c{/solr,null,AVAILABLE}
   [junit4]   2> 554342 INFO  (jetty-launcher-749-thread-6) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@1b14cb3{/solr,null,AVAILABLE}
   [junit4]   2> 554343 INFO  (jetty-launcher-749-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2016-09-29T17:03:22.226318Z
   [junit4]   2> 554342 INFO  (jetty-launcher-749-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 554343 INFO  (jetty-launcher-749-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2016-09-29T17:03:22.226522Z
   [junit4]   2> 554343 INFO  (jetty-launcher-749-thread-3) [    ] 
o.e.j.s.ServerConnector Started 
ServerConnector@437519{HTTP/1.1,[http/1.1]}{127.0.0.1:37150}
   [junit4]   2> 554343 INFO  (jetty-launcher-749-thread-3) [    ] 
o.e.j.s.Server Started @556813ms
   [junit4]   2> 554343 INFO  (jetty-launcher-749-thread-6) [    ] 
o.e.j.s.ServerConnector Started 
ServerConnector@7abcbc{HTTP/1.1,[http/1.1]}{127.0.0.1:43606}
   [junit4]   2> 554343 INFO  (jetty-launcher-749-thread-3) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=37150}
   [junit4]   2> 554343 INFO  (jetty-launcher-749-thread-6) [    ] 
o.e.j.s.Server Started @556813ms
   [junit4]   2> 554343 INFO  (jetty-launcher-749-thread-6) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=43606}
   [junit4]   2> 554342 INFO  (jetty-launcher-749-thread-5) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@633ca7{/solr,null,AVAILABLE}
   [junit4]   2> 554344 INFO  (jetty-launcher-749-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 
7.0.0
   [junit4]   2> 554344 INFO  (jetty-launcher-749-thread-6) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 
7.0.0
   [junit4]   2> 554344 INFO  (jetty-launcher-749-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 554344 INFO  (jetty-launcher-749-thread-6) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 554344 INFO  (jetty-launcher-749-thread-6) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 554344 INFO  (jetty-launcher-749-thread-6) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2016-09-29T17:03:22.227330Z
   [junit4]   2> 554344 INFO  (jetty-launcher-749-thread-5) [    ] 
o.e.j.s.ServerConnector Started 
ServerConnector@156b965{HTTP/1.1,[http/1.1]}{127.0.0.1:44287}
   [junit4]   2> 554344 INFO  (jetty-launcher-749-thread-5) [    ] 
o.e.j.s.Server Started @556814ms
   [junit4]   2> 554344 INFO  (jetty-launcher-749-thread-5) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=44287}
   [junit4]   2> 554344 INFO  (jetty-launcher-749-thread-5) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 
7.0.0
   [junit4]   2> 554344 INFO  (jetty-launcher-749-thread-5) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 554344 INFO  (jetty-launcher-749-thread-5) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 554344 INFO  (jetty-launcher-749-thread-5) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2016-09-29T17:03:22.227747Z
   [junit4]   2> 554344 INFO  (jetty-launcher-749-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 554344 INFO  (jetty-launcher-749-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2016-09-29T17:03:22.227912Z
   [junit4]   2> 554347 INFO  (jetty-launcher-749-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 554347 INFO  (jetty-launcher-749-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 554356 INFO  (jetty-launcher-749-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 554356 INFO  (jetty-launcher-749-thread-5) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 554356 INFO  (jetty-launcher-749-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 554357 INFO  (jetty-launcher-749-thread-6) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 554358 INFO  (jetty-launcher-749-thread-2) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34578/solr
   [junit4]   2> 554361 INFO  (jetty-launcher-749-thread-1) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34578/solr
   [junit4]   2> 554361 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from 
client sessionid 0x15776e688000005, likely client has closed socket
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2>        at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
   [junit4]   2> 554362 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from 
client sessionid 0x15776e688000006, likely client has closed socket
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2>        at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
   [junit4]   2> 554362 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from 
client sessionid 0x15776e688000004, likely client has closed socket
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2>        at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
   [junit4]   2> 554362 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from 
client sessionid 0x15776e688000002, likely client has closed socket
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2>        at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
   [junit4]   2> 554364 INFO  (jetty-launcher-749-thread-6) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34578/solr
   [junit4]   2> 554365 INFO  (jetty-launcher-749-thread-3) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34578/solr
   [junit4]   2> 554371 INFO  (jetty-launcher-749-thread-5) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34578/solr
   [junit4]   2> 554374 INFO  (jetty-launcher-749-thread-4) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34578/solr
   [junit4]   2> 554376 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from 
client sessionid 0x15776e688000008, likely client has closed socket
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2>        at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
   [junit4]   2> 554378 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from 
client sessionid 0x15776e688000009, likely client has closed socket
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2>        at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
   [junit4]   2> 554378 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from 
client sessionid 0x15776e68800000b, likely client has closed socket
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2>        at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
   [junit4]   2> 554379 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from 
client sessionid 0x15776e68800000a, likely client has closed socket
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2>        at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
   [junit4]   2> 554393 INFO  (jetty-launcher-749-thread-6) [    ] 
o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:43606_solr
   [junit4]   2> 554393 INFO  (jetty-launcher-749-thread-4) [    ] 
o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:37968_solr
   [junit4]   2> 554394 INFO  (jetty-launcher-749-thread-2) [    ] 
o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:32944_solr
   [junit4]   2> 554394 INFO  (jetty-launcher-749-thread-3) [    ] 
o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:37150_solr
   [junit4]   2> 554394 INFO  (jetty-launcher-749-thread-5) [    ] 
o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:44287_solr
   [junit4]   2> 554394 INFO  (jetty-launcher-749-thread-6) [    ] 
o.a.s.c.Overseer Overseer 
(id=96676649508012049-127.0.0.1:43606_solr-n_0000000000) starting
   [junit4]   2> 554394 INFO  (zkCallback-778-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 554394 INFO  (zkCallback-777-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 554394 INFO  (zkCallback-769-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 554394 INFO  (zkCallback-780-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 554394 INFO  (zkCallback-779-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 554395 INFO  (jetty-launcher-749-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 554395 INFO  (zkCallback-778-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (4)
   [junit4]   2> 554395 INFO  (zkCallback-780-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (4)
   [junit4]   2> 554395 INFO  (zkCallback-769-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (4)
   [junit4]   2> 554395 INFO  (zkCallback-779-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (4)
   [junit4]   2> 554395 INFO  (zkCallback-777-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (4)
   [junit4]   2> 554396 INFO  (zkCallback-776-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 554398 INFO  (jetty-launcher-749-thread-1) [    ] 
o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:42537_solr
   [junit4]   2> 554399 INFO  (zkCallback-779-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 554399 INFO  (zkCallback-776-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 554399 INFO  (zkCallback-777-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 554399 INFO  (zkCallback-780-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 554400 INFO  (jetty-launcher-749-thread-6) [    ] 
o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:43606_solr
   [junit4]   2> 554403 INFO  (zkCallback-769-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 554403 INFO  (zkCallback-776-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 554403 INFO  (zkCallback-779-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 554403 INFO  (zkCallback-780-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 554403 INFO  (zkCallback-769-thread-2) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 554412 INFO  (zkCallback-777-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 554414 INFO  (jetty-launcher-749-thread-4) [    ] 
o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath 
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteNodeTest_8313F26FCB3FBD32-001/tempDir-001/node4/.
   [junit4]   2> 554421 INFO  (jetty-launcher-749-thread-3) [    ] 
o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath 
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteNodeTest_8313F26FCB3FBD32-001/tempDir-001/node3/.
   [junit4]   2> 554422 INFO  (zkCallback-778-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (6)
   [junit4]   2> 554429 INFO  (jetty-launcher-749-thread-2) [    ] 
o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath 
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteNodeTest_8313F26FCB3FBD32-001/tempDir-001/node2/.
   [junit4]   2> 554436 INFO  (jetty-launcher-749-thread-5) [    ] 
o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath 
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteNodeTest_8313F26FCB3FBD32-001/tempDir-001/node5/.
   [junit4]   2> 554439 INFO  (jetty-launcher-749-thread-1) [    ] 
o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath 
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteNodeTest_8313F26FCB3FBD32-001/tempDir-001/node1/.
   [junit4]   2> 554446 INFO  (jetty-launcher-749-thread-6) [    ] 
o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath 
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteNodeTest_8313F26FCB3FBD32-001/tempDir-001/node6/.
   [junit4]   2> 554451 INFO  
(SUITE-DeleteNodeTest-seed#[8313F26FCB3FBD32]-worker) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (6)
   [junit4]   2> 554493 INFO  
(TEST-DeleteNodeTest.test-seed#[8313F26FCB3FBD32]) [    ] o.a.s.SolrTestCaseJ4 
###Starting test
   [junit4]   2> 554498 INFO  (qtp25347926-3750) [    ] 
o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params 
replicationFactor=2&collection.configName=conf1&maxShardsPerNode=3&name=deletenodetest_coll&action=CREATE&numShards=5&createNodeSet=127.0.0.1:43606_solr,127.0.0.1:42537_solr,127.0.0.1:32944_solr,127.0.0.1:37150_solr,127.0.0.1:37968_solr,127.0.0.1:44287_solr&wt=javabin&version=2
 and sendToOCPQueue=true
   [junit4]   2> 554499 INFO  (OverseerThreadFactory-1608-thread-1) [    ] 
o.a.s.c.CreateCollectionCmd Create collection deletenodetest_coll
   [junit4]   2> 554604 INFO  (qtp13527231-3721) [    ] 
o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&collection.configName=conf1&name=deletenodetest_coll_shard4_replica1&action=CREATE&numShards=5&collection=deletenodetest_coll&shard=shard4&wt=javabin&version=2
   [junit4]   2> 554605 INFO  (qtp25347926-3756) [    ] 
o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&collection.configName=conf1&name=deletenodetest_coll_shard4_replica2&action=CREATE&numShards=5&collection=deletenodetest_coll&shard=shard4&wt=javabin&version=2
   [junit4]   2> 554605 INFO  (qtp22725655-3760) [    ] 
o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&collection.configName=conf1&name=deletenodetest_coll_shard2_replica2&action=CREATE&numShards=5&collection=deletenodetest_coll&shard=shard2&wt=javabin&version=2
   [junit4]   2> 554606 INFO  (qtp22725655-3736) [    ] 
o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&collection.configName=conf1&name=deletenodetest_coll_shard5_replica2&action=CREATE&numShards=5&collection=deletenodetest_coll&shard=shard5&wt=javabin&version=2
   [junit4]   2> 554606 INFO  (qtp25187729-3744) [    ] 
o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&collection.configName=conf1&name=deletenodetest_coll_shard5_replica1&action=CREATE&numShards=5&collection=deletenodetest_coll&shard=shard5&wt=javabin&version=2
   [junit4]   2> 554606 INFO  (qtp20049145-3764) [    ] 
o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&collection.configName=conf1&name=deletenodetest_coll_shard3_replica1&action=CREATE&numShards=5&collection=deletenodetest_coll&shard=shard3&wt=javabin&version=2
   [junit4]   2> 554607 INFO  (qtp25187729-3717) [    ] 
o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&collection.configName=conf1&name=deletenodetest_coll_shard2_replica1&action=CREATE&numShards=5&collection=deletenodetest_coll&shard=shard2&wt=javabin&version=2
   [junit4]   2> 554607 INFO  (qtp13527231-3748) [    ] 
o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&collection.configName=conf1&name=deletenodetest_coll_shard1_replica1&action=CREATE&numShards=5&collection=deletenodetest_coll&shard=shard1&wt=javabin&version=2
   [junit4]   2> 554610 INFO  (qtp12370062-3729) [    ] 
o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&collection.configName=conf1&name=deletenodetest_coll_shard3_replica2&action=CREATE&numShards=5&collection=deletenodetest_coll&shard=shard3&wt=javabin&version=2
   [junit4]   2> 554610 INFO  (qtp25347926-3720) [    ] 
o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&collection.configName=conf1&name=deletenodetest_coll_shard1_replica2&action=CREATE&numShards=5&collection=deletenodetest_coll&shard=shard1&wt=javabin&version=2
   [junit4]   2> 554722 INFO  (zkCallback-778-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/deletenodetest_coll/state.json] for collection 
[deletenodetest_coll] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 554722 INFO  (zkCallback-776-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/deletenodetest_coll/state.json] for collection 
[deletenodetest_coll] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 554722 INFO  (zkCallback-769-thread-2) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/deletenodetest_coll/state.json] for collection 
[deletenodetest_coll] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 554722 INFO  (zkCallback-780-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/deletenodetest_coll/state.json] for collection 
[deletenodetest_coll] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 554722 INFO  (zkCallback-777-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/deletenodetest_coll/state.json] for collection 
[deletenodetest_coll] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 554722 INFO  (zkCallback-779-thread-3) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/deletenodetest_coll/state.json] for collection 
[deletenodetest_coll] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 555616 INFO  (qtp13527231-3748) [    ] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 555617 INFO  (qtp25187729-3717) [    ] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 555617 INFO  (qtp25187729-3744) [    ] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 555617 INFO  (qtp13527231-3721) [    ] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 555617 INFO  (qtp20049145-3764) [    ] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 555617 INFO  (qtp25347926-3756) [    ] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 555635 INFO  (qtp12370062-3729) [    ] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 555636 INFO  (qtp25347926-3720) [    ] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 555638 INFO  (qtp13527231-3748) [    ] o.a.s.s.IndexSchema 
[deletenodetest_coll_shard1_replica1] Schema name=cloud-dynamic
   [junit4]   2> 555638 INFO  (qtp25187729-3717) [    ] o.a.s.s.IndexSchema 
[deletenodetest_coll_shard2_replica1] Schema name=cloud-dynamic
   [junit4]   2> 555638 INFO  (qtp25187729-3744) [    ] o.a.s.s.IndexSchema 
[deletenodetest_coll_shard5_replica1] Schema name=cloud-dynamic
   [junit4]   2> 555639 INFO  (qtp22725655-3760) [    ] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 555651 INFO  (qtp22725655-3736) [    ] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 555652 INFO  (qtp13527231-3721) [    ] o.a.s.s.IndexSchema 
[deletenodetest_coll_shard4_replica1] Schema name=cloud-dynamic
   [junit4]   2> 555654 INFO  (qtp25347926-3720) [    ] o.a.s.s.IndexSchema 
[deletenodetest_coll_shard1_replica2] Schema name=cloud-dynamic
   [junit4]   2> 555654 INFO  (qtp25347926-3756) [    ] o.a.s.s.IndexSchema 
[deletenodetest_coll_shard4_replica2] Schema name=cloud-dynamic
   [junit4]   2> 555657 INFO  (qtp22725655-3760) [    ] o.a.s.s.IndexSchema 
[deletenodetest_coll_shard2_replica2] Schema name=cloud-dynamic
   [junit4]   2> 555657 INFO  (qtp20049145-3764) [    ] o.a.s.s.IndexSchema 
[deletenodetest_coll_shard3_replica1] Schema name=cloud-dynamic
   [junit4]   2> 555667 INFO  (qtp12370062-3729) [    ] o.a.s.s.IndexSchema 
[deletenodetest_coll_shard3_replica2] Schema name=cloud-dynamic
   [junit4]   2> 555672 WARN  (qtp25187729-3744) [    ] o.a.s.s.IndexSchema 
[deletenodetest_coll_shard5_replica1] default search field in schema is text. 
WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 555679 INFO  (qtp25187729-3744) [    ] o.a.s.s.IndexSchema 
Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 555679 INFO  (qtp25187729-3744) [    ] o.a.s.c.CoreContainer 
Creating SolrCore 'deletenodetest_coll_shard5_replica1' using configuration 
from collection deletenodetest_coll
   [junit4]   2> 555680 WARN  (qtp13527231-3748) [    ] o.a.s.s.IndexSchema 
[deletenodetest_coll_shard1_replica1] default search field in schema is text. 
WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 555680 INFO  (qtp25187729-3744) [    ] o.a.s.c.SolrCore 
[[deletenodetest_coll_shard5_replica1] ] Opening new SolrCore at 
[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteNodeTest_8313F26FCB3FBD32-001/tempDir-001/node4/deletenodetest_coll_shard5_replica1],
 
dataDir=[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteNodeTest_8313F26FCB3FBD32-001/tempDir-001/node4/./deletenodetest_coll_shard5_replica1/data/]
   [junit4]   2> 555689 WARN  (qtp13527231-3721) [    ] o.a.s.s.IndexSchema 
[deletenodetest_coll_shard4_replica1] default search field in schema is text. 
WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 555690 INFO  (qtp13527231-3721) [    ] o.a.s.s.IndexSchema 
Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 555690 INFO  (qtp22725655-3736) [    ] o.a.s.s.IndexSchema 
[deletenodetest_coll_shard5_replica2] Schema name=cloud-dynamic
   [junit4]   2> 555690 INFO  (qtp13527231-3721) [    ] o.a.s.c.CoreContainer 
Creating SolrCore 'deletenodetest_coll_shard4_replica1' using configuration 
from collection deletenodetest_coll
   [junit4]   2> 555691 INFO  (qtp13527231-3721) [    ] o.a.s.c.SolrCore 
[[deletenodetest_coll_shard4_replica1] ] Opening new SolrCore at 
[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteNodeTest_8313F26FCB3FBD32-001/tempDir-001/node3/deletenodetest_coll_shard4_replica1],
 
dataDir=[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteNodeTest_8313F26FCB3FBD32-001/tempDir-001/node3/./deletenodetest_coll_shard4_replica1/data/]
   [junit4]   2> 555692 WARN  (qtp25347926-3756) [    ] o.a.s.s.IndexSchema 
[deletenodetest_coll_shard4_replica2] default search field in schema is text. 
WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 555693 INFO  (qtp25187729-3744) [    ] o.a.s.u.UpdateHandler 
Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 555693 INFO  (qtp25187729-3744) [    ] o.a.s.u.UpdateLog 
Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 555694 INFO  (qtp25347926-3756) [    ] o.a.s.s.IndexSchema 
Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 555694 INFO  (qtp25187729-3744) [    ] o.a.s.u.CommitTracker 
Hard AutoCommit: disabled
   [junit4]   2> 555694 INFO  (qtp25187729-3744) [    ] o.a.s.u.CommitTracker 
Soft AutoCommit: disabled
   [junit4]   2> 555694 INFO  (qtp25187729-3744) [    ] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@194b105[deletenodetest_coll_shard5_replica1] main]
   [junit4]   2> 555694 INFO  (qtp25347926-3756) [    ] o.a.s.c.CoreContainer 
Creating SolrCore 'deletenodetest_coll_shard4_replica2' using configuration 
from collection deletenodetest_coll
   [junit4]   2> 555695 INFO  (qtp25187729-3744) [    ] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf1
   [junit4]   2> 555695 INFO  (qtp25187729-3744) [    ] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 555695 INFO  (qtp25347926-3756) [    ] o.a.s.c.SolrCore 
[[deletenodetest_coll_shard4_replica2] ] Opening new SolrCore at 
[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteNodeTest_8313F26FCB3FBD32-001/tempDir-001/node1/deletenodetest_coll_shard4_replica2],
 
dataDir=[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteNodeTest_8313F26FCB3FBD32-001/tempDir-001/node1/./deletenodetest_coll_shard4_replica2/data/]
   [junit4]   2> 555695 INFO  (qtp25187729-3744) [    ] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 555695 INFO  (qtp25187729-3744) [    ] o.a.s.u.UpdateLog Could 
not find max version in index or recent updates, using new clock 
1546826393665404928
   [junit4]   2> 555698 INFO  (searcherExecutor-1621-thread-1) [    ] 
o.a.s.c.SolrCore [deletenodetest_coll_shard5_replica1] Registered new searcher 
Searcher@194b105[deletenodetest_coll_shard5_replica1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 555698 INFO  (qtp13527231-3721) [    ] o.a.s.u.UpdateHandler 
Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 555699 INFO  (qtp13527231-3721) [    ] o.a.s.u.UpdateLog 
Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 555700 INFO  (qtp13527231-3721) [    ] o.a.s.u.CommitTracker 
Hard AutoCommit: disabled
   [junit4]   2> 555700 INFO  (qtp13527231-3721) [    ] o.a.s.u.CommitTracker 
Soft AutoCommit: disabled
   [junit4]   2> 555700 INFO  (qtp13527231-3721) [    ] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@1eebf50[deletenodetest_coll_shard4_replica1] main]
   [junit4]   2> 555701 WARN  (qtp20049145-3764) [    ] o.a.s.s.IndexSchema 
[deletenodetest_coll_shard3_replica1] default search field in schema is text. 
WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 555702 INFO  (qtp20049145-3764) [    ] o.a.s.s.IndexSchema 
Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 555703 WARN  (qtp25187729-3717) [    ] o.a.s.s.IndexSchema 
[deletenodetest_coll_shard2_replica1] default search field in schema is text. 
WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 555703 WARN  (qtp22725655-3760) [    ] o.a.s.s.IndexSchema 
[deletenodetest_coll_shard2_replica2] default search field in schema is text. 
WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 555703 INFO  (qtp20049145-3764) [    ] o.a.s.c.CoreContainer 
Creating SolrCore 'deletenodetest_coll_shard3_replica1' using configuration 
from collection deletenodetest_coll
   [junit4]   2> 555703 INFO  (qtp25187729-3717) [    ] o.a.s.s.IndexSchema 
Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 555704 INFO  (qtp22725655-3760) [    ] o.a.s.s.IndexSchema 
Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 555704 INFO  (qtp20049145-3764) [    ] o.a.s.c.SolrCore 
[[deletenodetest_coll_shard3_replica1] ] Opening new SolrCore at 
[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteNodeTest_8313F26FCB3FBD32-001/tempDir-001/node6/deletenodetest_coll_shard3_replica1],
 
dataDir=[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteNodeTest_8313F26FCB3FBD32-001/tempDir-001/node6/./deletenodetest_coll_shard3_replica1/data/]
   [junit4]   2> 555704 INFO  (qtp13527231-3748) [    ] o.a.s.s.IndexSchema 
Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 555704 INFO  (qtp22725655-3760) [    ] o.a.s.c.CoreContainer 
Creating SolrCore 'deletenodetest_coll_shard2_replica2' using configuration 
from collection deletenodetest_coll
   [junit4]   2> 555704 INFO  (qtp25187729-3717) [    ] o.a.s.c.CoreContainer 
Creating SolrCore 'deletenodetest_coll_shard2_replica1' using configuration 
from collection deletenodetest_coll
   [junit4]   2> 555705 INFO  (qtp22725655-3760) [    ] o.a.s.c.SolrCore 
[[deletenodetest_coll_shard2_replica2] ] Opening new SolrCore at 
[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteNodeTest_8313F26FCB3FBD32-001/tempDir-001/node5/deletenodetest_coll_shard2_replica2],
 
dataDir=[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteNodeTest_8313F26FCB3FBD32-001/tempDir-001/node5/./deletenodetest_coll_shard2_replica2/data/]
   [junit4]   2> 555705 INFO  (qtp25187729-3717) [    ] o.a.s.c.SolrCore 
[[deletenodetest_coll_shard2_replica1] ] Opening new SolrCore at 
[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteNodeTest_8313F26FCB3FBD32-001/tempDir-001/node4/deletenodetest_coll_shard2_replica1],
 
dataDir=[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteNodeTest_8313F26FCB3FBD32-001/tempDir-001/node4/./deletenodetest_coll_shard2_replica1/data/]
   [junit4]   2> 555705 INFO  (qtp25347926-3756) [    ] o.a.s.u.UpdateHandler 
Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 555705 INFO  (qtp25347926-3756) [    ] o.a.s.u.UpdateLog 
Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 555713 INFO  (qtp13527231-3721) [    ] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf1
   [junit4]   2> 555714 INFO  (qtp25347926-3756) [    ] o.a.s.u.CommitTracker 
Hard AutoCommit: disabled
   [junit4]   2> 555714 INFO  (qtp25347926-3756) [    ] o.a.s.u.CommitTracker 
Soft AutoCommit: disabled
   [junit4]   2> 555714 INFO  (qtp13527231-3721) [    ] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 555714 INFO  (qtp13527231-3721) [    ] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 555714 INFO  (qtp25347926-3756) [    ] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@1e6dad6[deletenodetest_coll_shard4_replica2] main]
   [junit4]   2> 555714 INFO  (qtp13527231-3748) [    ] o.a.s.c.CoreContainer 
Creating SolrCore 'deletenodetest_coll_shard1_replica1' using configuration 
from collection deletenodetest_coll
   [junit4]   2> 555714 INFO  (searcherExecutor-1622-thread-1) [    ] 
o.a.s.c.SolrCore [deletenodetest_coll_shard4_replica1] Registered new searcher 
Searcher@1eebf50[deletenodetest_coll_shard4_replica1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 555714 INFO  (qtp13527231-3721) [    ] o.a.s.u.UpdateLog Could 
not find max version in index or recent updates, using new clock 
1546826393685327872
   [junit4]   2> 555715 INFO  (qtp13527231-3748) [    ] o.a.s.c.SolrCore 
[[deletenodetest_coll_shard1_replica1] ] Opening new SolrCore at 
[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteNodeTest_8313F26FCB3FBD32-001/tempDir-001/node3/deletenodetest_coll_shard1_replica1],
 
dataDir=[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteNodeTest_8313F26FCB3FBD32-001/tempDir-001/node3/./deletenodetest_coll_shard1_replica1/data/]
   [junit4]   2> 555715 INFO  (qtp25347926-3756) [    ] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf1
   [junit4]   2> 555716 WARN  (qtp25347926-3720) [    ] o.a.s.s.IndexSchema 
[deletenodetest_coll_shard1_replica2] default search field in schema is text. 
WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 555716 INFO  (qtp25347926-3756) [    ] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 555716 INFO  (qtp25347926-3756) [    ] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 555716 INFO  (qtp25347926-3756) [    ] o.a.s.u.UpdateLog Could 
not find max version in index or recent updates, using new clock 
1546826393687425024
   [junit4]   2> 555717 WARN  (qtp12370062-3729) [    ] o.a.s.s.IndexSchema 
[deletenodetest_coll_shard3_replica2] default search field in schema is text. 
WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 555717 INFO  (qtp25187729-3744) [    ] 
o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for 
shard shard5: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 555717 INFO  (searcherExecutor-1627-thread-1) [    ] 
o.a.s.c.SolrCore [deletenodetest_coll_shard4_replica2] Registered new searcher 
Searcher@1e6dad6[deletenodetest_coll_shard4_replica2] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 555717 INFO  (qtp25347926-3720) [    ] o.a.s.s.IndexSchema 
Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 555718 INFO  (qtp12370062-3729) [    ] o.a.s.s.IndexSchema 
Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 555718 WARN  (qtp22725655-3736) [    ] o.a.s.s.IndexSchema 
[deletenodetest_coll_shard5_replica2] default search field in schema is text. 
WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 555718 INFO  (qtp12370062-3729) [    ] o.a.s.c.CoreContainer 
Creating SolrCore 'deletenodetest_coll_shard3_replica2' using configuration 
from collection deletenodetest_coll
   [junit4]   2> 555719 INFO  (qtp12370062-3729) [    ] o.a.s.c.SolrCore 
[[deletenodetest_coll_shard3_replica2] ] Opening new SolrCore at 
[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteNodeTest_8313F26FCB3FBD32-001/tempDir-001/node2/deletenodetest_coll_shard3_replica2],
 
dataDir=[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteNodeTest_8313F26FCB3FBD32-001/tempDir-001/node2/./deletenodetest_coll_shard3_replica2/data/]
   [junit4]   2> 555719 INFO  (qtp22725655-3736) [    ] o.a.s.s.IndexSchema 
Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 555719 INFO  (qtp22725655-3736) [    ] o.a.s.c.CoreContainer 
Creating SolrCore 'deletenodetest_coll_shard5_replica2' using configuration 
from collection deletenodetest_coll
   [junit4]   2> 555720 INFO  (qtp22725655-3736) [    ] o.a.s.c.SolrCore 
[[deletenodetest_coll_shard5_replica2] ] Opening new SolrCore at 
[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteNodeTest_8313F26FCB3FBD32-001/tempDir-001/node5/deletenodetest_coll_shard5_replica2],
 
dataDir=[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteNodeTest_8313F26FCB3FBD32-001/tempDir-001/node5/./deletenodetest_coll_shard5_replica2/data/]
   [junit4]   2> 555720 INFO  (qtp25347926-3720) [    ] o.a.s.c.CoreContainer 
Creating SolrCore 'deletenodetest_coll_shard1_replica2' using configuration 
from collection deletenodetest_coll
   [junit4]   2> 555720 INFO  (qtp25347926-3756) [    ] 
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 555720 INFO  (qtp25347926-3756) [    ] 
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 555720 INFO  (qtp25347926-3756) [    ] o.a.s.c.SyncStrategy 
Sync replicas to 
http://127.0.0.1:42537/solr/deletenodetest_coll_shard4_replica2/
   [junit4]   2> 555720 INFO  (qtp25347926-3756) [    ] o.a.s.u.PeerSync 
PeerSync: core=deletenodetest_coll_shard4_replica2 
url=http://127.0.0.1:42537/solr START 
replicas=[http://127.0.0.1:37150/solr/deletenodetest_coll_shard4_replica1/] 
nUpdates=100
   [junit4]   2> 555721 INFO  (qtp25347926-3720) [    ] o.a.s.c.SolrCore 
[[deletenodetest_coll_shard1_replica2] ] Opening new SolrCore at 
[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteNodeTest_8313F26FCB3FBD32-001/tempDir-001/node1/deletenodetest_coll_shard1_replica2],
 
dataDir=[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteNodeTest_8313F26FCB3FBD32-001/tempDir-001/node1/./deletenodetest_coll_shard1_replica2/data/]
   [junit4]   2> 555722 INFO  (qtp25347926-3756) [    ] o.a.s.u.PeerSync 
PeerSync: core=deletenodetest_coll_shard4_replica2 
url=http://127.0.0.1:42537/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 555722 INFO  (qtp20049145-3764) [    ] o.a.s.u.UpdateHandler 
Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 555722 INFO  (qtp20049145-3764) [    ] o.a.s.u.UpdateLog 
Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 555724 INFO  (qtp20049145-3764) [    ] o.a.s.u.CommitTracker 
Hard AutoCommit: disabled
   [junit4]   2> 555724 INFO  (qtp20049145-3764) [    ] o.a.s.u.CommitTracker 
Soft AutoCommit: disabled
   [junit4]   2> 555724 INFO  (qtp25187729-3717) [    ] o.a.s.u.UpdateHandler 
Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 555724 INFO  (qtp25187729-3717) [    ] o.a.s.u.UpdateLog 
Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 555724 INFO  (qtp20049145-3764) [    ] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@16876ab[deletenodetest_coll_shard3_replica1] main]
   [junit4]   2> 555725 INFO  (qtp25187729-3717) [    ] o.a.s.u.CommitTracker 
Hard AutoCommit: disabled
   [junit4]   2> 555725 INFO  (qtp25187729-3717) [    ] o.a.s.u.CommitTracker 
Soft AutoCommit: disabled
   [junit4]   2> 555725 INFO  (qtp20049145-3764) [    ] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf1
   [junit4]   2> 555725 INFO  (qtp25187729-3717) [    ] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@8271d6[deletenodetest_coll_shard2_replica1] main]
   [junit4]   2> 555726 INFO  (qtp20049145-3764) [    ] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 555726 INFO  (qtp20049145-3764) [    ] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 555726 INFO  (searcherExecutor-1632-thread-1) [    ] 
o.a.s.c.SolrCore [deletenodetest_coll_shard3_replica1] Registered new searcher 
Searcher@16876ab[deletenodetest_coll_shard3_replica1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 555726 INFO  (qtp20049145-3764) [    ] o.a.s.u.UpdateLog Could 
not find max version in index or recent updates, using new clock 
1546826393697910784
   [junit4]   2> 555727 INFO  (qtp25187729-3717) [    ] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf1
   [junit4]   2> 555727 INFO  (qtp25187729-3717) [    ] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 555727 INFO  (qtp25187729-3717) [    ] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 555727 INFO  (qtp22725655-3760) [    ] o.a.s.u.UpdateHandler 
Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 555727 INFO  (qtp22725655-3760) [    ] o.a.s.u.UpdateLog 
Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 555727 INFO  (searcherExecutor-1635-thread-1) [    ] 
o.a.s.c.SolrCore [deletenodetest_coll_shard2_replica1] Registered new searcher 
Searcher@8271d6[deletenodetest_coll_shard2_replica1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 555727 INFO  (qtp25187729-3717) [    ] o.a.s.u.UpdateLog Could 
not find max version in index or recent updates, using new clock 
1546826393698959360
   [junit4]   2> 555728 INFO  (qtp12370062-3729) [    ] o.a.s.u.UpdateHandler 
Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 555728 INFO  (qtp12370062-3729) [    ] o.a.s.u.UpdateLog 
Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 555728 INFO  (qtp22725655-3760) [    ] o.a.s.u.CommitTracker 
Hard AutoCommit: disabled
   [junit4]   2> 555728 INFO  (qtp22725655-3760) [    ] o.a.s.u.CommitTracker 
Soft AutoCommit: disabled
   [junit4]   2> 555729 INFO  (qtp12370062-3729) [    ] o.a.s.u.CommitTracker 
Hard AutoCommit: disabled
   [junit4]   2> 555729 INFO  (qtp12370062-3729) [    ] o.a.s.u.CommitTracker 
Soft AutoCommit: disabled
   [junit4]   2> 555729 INFO  (qtp22725655-3760) [    ] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@10e665b[deletenodetest_coll_shard2_replica2] main]
   [junit4]   2> 555729 INFO  (qtp12370062-3729) [    ] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@1bc6180[deletenodetest_coll_shard3_replica2] main]
   [junit4]   2> 555730 INFO  (qtp22725655-3760) [    ] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf1
   [junit4]   2> 555730 INFO  (qtp22725655-3760) [    ] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 555730 INFO  (qtp22725655-3760) [    ] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 555731 INFO  (searcherExecutor-1634-thread-1) [    ] 
o.a.s.c.SolrCore [deletenodetest_coll_shard2_replica2] Registered new searcher 
Searcher@10e665b[deletenodetest_coll_shard2_replica2] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 555732 INFO  (qtp13527231-3748) [    ] o.a.s.u.UpdateHandler 
Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 555734 INFO  (qtp13527231-3748) [    ] o.a.s.u.UpdateLog 
Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 555734 INFO  (qtp22725655-3760) [    ] o.a.s.u.UpdateLog Could 
not find max version in index or recent updates, using new clock 
1546826393706299392
   [junit4]   2> 555734 INFO  (qtp12370062-3729) [    ] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf1
   [junit4]   2> 555735 INFO  (qtp12370062-3729) [    ] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 555735 INFO  (qtp12370062-3729) [    ] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 555735 INFO  (searcherExecutor-1640-thread-1) [    ] 
o.a.s.c.SolrCore [deletenodetest_coll_shard3_replica2] Registered new searcher 
Searcher@1bc6180[deletenodetest_coll_shard3_replica2] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 555735 INFO  (qtp12370062-3729) [    ] o.a.s.u.UpdateLog Could 
not find max version in index or recent updates, using new clock 
1546826393707347968
   [junit4]   2> 555735 INFO  (qtp13527231-3748) [    ] o.a.s.u.CommitTracker 
Hard AutoCommit: disabled
   [junit4]   2> 555735 INFO  (qtp13527231-3748) [    ] o.a.s.u.CommitTracker 
Soft AutoCommit: disabled
   [junit4]   2> 555731 INFO  (qtp13527231-3719) [    ] o.a.s.c.S.Request 
[deletenodetest_coll_shard4_replica1]  webapp=/solr path=/get 
params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2}
 status=0 QTime=1
   [junit4]   2> 555736 INFO  (qtp13527231-3748) [    ] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@1e21fcc[deletenodetest_coll_shard1_replica1] main]
   [junit4]   2> 555736 INFO  (qtp25347926-3720) [    ] o.a.s.u.UpdateHandler 
Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 555736 INFO  (qtp25347926-3720) [    ] o.a.s.u.UpdateLog 
Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 555736 INFO  (qtp25187729-3717) [    ] 
o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for 
shard shard2: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 555736 INFO  (qtp13527231-3748) [    ] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf1
   [junit4]   2> 555737 INFO  (qtp13527231-3748) [    ] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 555737 INFO  (qtp13527231-3748) [    ] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 555737 INFO  (qtp25347926-3720) [    ] o.a.s.u.CommitTracker 
Hard AutoCommit: disabled
   [junit4]   2> 555737 INFO  (qtp25347926-3720) [    ] o.a.s.u.CommitTracker 
Soft AutoCommit: disabled
   [junit4]   2> 555737 INFO  (qtp13527231-3748) [    ] o.a.s.u.UpdateLog Could 
not find max version in index or recent updates, using new clock 
1546826393709445120
   [junit4]   2> 555737 INFO  (qtp25347926-3720) [    ] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@1a687ec[deletenodetest_coll_shard1_replica2] main]
   [junit4]   2> 555738 INFO  (qtp20049145-3764) [    ] 
o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for 
shard shard3: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 555739 INFO  (qtp25347926-3720) [    ] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf1
   [junit4]   2> 555739 INFO  (qtp25347926-3720) [    ] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 555739 INFO  (qtp25347926-3720) [    ] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 555739 INFO  (searcherExecutor-1643-thread-1) [    ] 
o.a.s.c.SolrCore [deletenodetest_coll_shard1_replica2] Registered new searcher 
Searcher@1a687ec[deletenodetest_coll_shard1_replica2] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 555739 INFO  (qtp25347926-3720) [    ] o.a.s.u.UpdateLog Could 
not find max version in index or recent updates, using new clock 
1546826393711542272
   [junit4]   2> 555739 INFO  (searcherExecutor-1639-thread-1) [    ] 
o.a.s.c.SolrCore [deletenodetest_coll_shard1_replica1] Registered new searcher 
Searcher@1e21fcc[deletenodetest_coll_shard1_replica1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 555739 INFO  (qtp22725655-3736) [    ] o.a.s.u.UpdateHandler 
Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 555740 INFO  (qtp22725655-3736) [    ] o.a.s.u.UpdateLog 
Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 555741 INFO  (qtp22725655-3736) [    ] o.a.s.u.CommitTracker 
Hard AutoCommit: disabled
   [junit4]   2> 555741 INFO  (qtp22725655-3736) [    ] o.a.s.u.CommitTracker 
Soft AutoCommit: disabled
   [junit4]   2> 555741 INFO  (qtp22725655-3736) [    ] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@1d4ec97[deletenodetest_coll_shard5_replica2] main]
   [junit4]   2> 555742 INFO  (qtp22725655-3736) [    ] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf1
   [junit4]   2> 555742 INFO  (qtp22725655-3736) [    ] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 555742 INFO  (qtp22725655-3736) [    ] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 555742 INFO  (searcherExecutor-1641-thread-1) [    ] 
o.a.s.c.SolrCore [deletenodetest_coll_shard5_replica2] Registered new searcher 
Searcher@1d4ec97[deletenodetest_coll_shard5_replica2] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 555742 INFO  (qtp22725655-3736) [    ] o.a.s.u.UpdateLog Could 
not find max version in index or recent updates, using new clock 
1546826393714688000
   [junit4]   2> 555745 INFO  (qtp13527231-3748) [    ] 
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 555745 INFO  (qtp13527231-3748) [    ] 
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 555745 INFO  (qtp13527231-3748) [    ] o.a.s.c.SyncStrategy 
Sync replicas to 
http://127.0.0.1:37150/solr/deletenodetest_coll_shard1_replica1/
   [junit4]   2> 555746 INFO  (qtp13527231-3748) [    ] o.a.s.u.PeerSync 
PeerSync: core=deletenodetest_coll_shard1_replica1 
url=http://127.0.0.1:37150/solr START 
replicas=[http://127.0.0.1:42537/solr/deletenodetest_coll_shard1_replica2/] 
nUpdates=100
   [junit4]   2> 555746 INFO  (qtp25347926-3756) [    ] o.a.s.c.SyncStrategy 
Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 555746 INFO  (qtp25347926-3756) [    ] 
o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we 
can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 555746 INFO  (qtp13527231-3748) [    ] o.a.s.u.PeerSync 
PeerSync: core=deletenodetest_coll_shard1_replica1 
url=http://127.0.0.1:37150/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 555748 INFO  (qtp25347926-3738) [    ] o.a.s.c.S.Request 
[deletenodetest_coll_shard1_replica2]  webapp=/solr path=/get 
params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2}
 status=0 QTime=0
   [junit4]   2> 555748 INFO  (qtp13527231-3748) [    ] o.a.s.c.SyncStrategy 
Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 555749 INFO  (qtp13527231-3748) [    ] 
o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we 
can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 555750 INFO  (qtp25347926-3756) [    ] 
o.a.s.c.ShardLeaderElectionContext I am the new leader: 
http://127.0.0.1:42537/solr/deletenodetest_coll_shard4_replica2/ shard4
   [junit4]   2> 555751 INFO  (qtp13527231-3748) [    ] 
o.a.s.c.ShardLeaderElectionContext I am the new leader: 
http://127.0.0.1:37150/solr/deletenodetest_coll_shard1_replica1/ shard1
   [junit4]   2> 555852 INFO  (zkCallback-777-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/deletenodetest_coll/state.json] for collection 
[deletenodetest_coll] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 555852 INFO  (zkCallback-769-thread-2) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/deletenodetest_coll/state.json] for collection 
[deletenodetest_coll] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 555852 INFO  (zkCallback-776-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/deletenodetest_coll/state.json] for collection 
[deletenodetest_coll] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 555852 INFO  (zkCallback-779-thread-2) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/deletenodetest_coll/state.json] for collection 
[deletenodetest_coll] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 555852 INFO  (zkCallback-778-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/deletenodetest_coll/state.json] for collection 
[deletenodetest_coll] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 555852 INFO  (zkCallback-780-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/deletenodetest_coll/state.json] for collection 
[deletenodetest_coll] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 555901 INFO  (qtp25347926-3756) [    ] o.a.s.c.ZkController I 
am the leader, no recovery necessary
   [junit4]   2> 555901 INFO  (qtp13527231-3748) [    ] o.a.s.c.ZkController I 
am the leader, no recovery necessary
   [junit4]   2> 555902 INFO  (qtp25347926-3756) [    ] o.a.s.s.HttpSolrCall 
[admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&collection.configName=conf1&name=deletenodetest_coll_shard4_replica2&action=CREATE&numShards=5&collection=deletenodetest_coll&shard=shard4&wt=javabin&version=2}
 status=0 QTime=1297
   [junit4]   2> 555902 INFO  (qtp13527231-3748) [    ] o.a.s.s.HttpSolrCall 
[admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&collection.configName=conf1&name=deletenodetest_coll_shard1_replica1&action=CREATE&numShards=5&collection=deletenodetest_coll&shard=shard1&wt=javabin&version=2}
 status=0 QTime=1295
   [junit4]   2> 556004 INFO  (zkCallback-777-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/deletenodetest_coll/state.json] for collection 
[deletenodetest_coll] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 556004 INFO  (zkCallback-778-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/deletenodetest_coll/state.json] for collection 
[deletenodetest_coll] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 556004 INFO  (zkCallback-776-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/deletenodetest_coll/state.json] for collection 
[deletenodetest_coll] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 556004 INFO  (zkCallback-779-thread-2) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/deletenodetest_coll/state.json] for collection 
[deletenodetest_coll] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 556004 INFO  (zkCallback-780-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/deletenodetest_coll/state.json] for collection 
[deletenodetest_coll] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 556004 INFO  (zkCallback-769-thread-2) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/deletenodetest_coll/state.json] for collection 
[deletenodetest_coll] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 556217 INFO  (qtp25187729-3744) [    ] 
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 556217 INFO  (qtp25187729-3744) [    ] 
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 556217 INFO  (qtp25187729-3744) [    ] o.a.s.c.SyncStrategy 
Sync replicas to 
http://127.0.0.1:37968/solr/deletenodetest_coll_shard5_replica1/
   [junit4]   2> 556217 INFO  (qtp25187729-3744) [    ] o.a.s.u.PeerSync 
PeerSync: core=deletenodetest_coll_shard5_replica1 
url=http://127.0.0.1:37968/solr START 
replicas=[http://127.0.0.1:44287/solr/deletenodetest_coll_shard5_replica2/] 
nUpdates=100
   [junit4]   2> 556218 INFO  (qtp25187729-3744) [    ] o.a.s.u.PeerSync 
PeerSync: core=deletenodetest_coll_shard5_replica1 
url=http://127.0.0.1:37968/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 556220 INFO  (qtp22725655-3734) [    ] o.a.s.c.S.Request 
[deletenodetest_coll_shard5_replica2]  webapp=/solr path=/get 
params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2}
 status=0 QTime=0
   [junit4]   2> 556220 INFO  (qtp25187729-3744) [    ] o.a.s.c.SyncStrategy 
Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 556220 INFO  (qtp25187729-3744) [    ] 
o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we 
can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 556222 INFO  (qtp25187729-3744) [    ] 
o.a.s.c.ShardLeaderElectionContext I am the new leader: 
http://127.0.0.1:37968/solr/deletenodetest_coll_shard5_replica1/ shard5
   [junit4]   2> 556237 INFO  (qtp25187729-3717) [    ] 
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 556237 INFO  (qtp25187729-3717) [    ] 
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 556237 INFO  (qtp25187729-3717) [    ] o.a.s.c.SyncStrategy 
Sync replicas to 
http://127.0.0.1:37968/solr/deletenodetest_coll_shard2_replica1/
   [junit4]   2> 556237 INFO  (qtp25187729-3717) [    ] o.a.s.u.PeerSync 
PeerSync: core=deletenodetest_coll_shard2_replica1 
url=http://127.0.0.1:37968/solr START 
replicas=[http://127.0.0.1:44287/solr/deletenodetest_coll_shard2_replica2/] 
nUpdates=100
   [junit4]   2> 556238 INFO  (qtp25187729-3717) [    ] o.a.s.u.PeerSync 
PeerSync: core=deletenodetest_coll_shard2_replica1 
url=http://127.0.0.1:37968/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 556238 INFO  (qtp20049145-3764) [    ] 
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 556238 INFO  (qtp20049145-3764) [    ] 
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 556238 INFO  (qtp20049145-3764) [    ] o.a.s.c.SyncStrategy 
Sync replicas to 
http://127.0.0.1:43606/solr/deletenodetest_coll_shard3_replica1/
   [junit4]   2> 556238 INFO  (qtp20049145-3764) [    ] o.a.s.u.PeerSync 
PeerSync: core=deletenodetest_coll_shard3_replica1 
url=http://127.0.0.1:43606/solr START 
replicas=[http://127.0.0.1:32944/solr/deletenodetest_coll_shard3_replica2/] 
nUpdates=100
   [junit4]   2> 556238 INFO  (qtp20049145-3764) [    ] o.a.s.u.PeerSync 
PeerSync: core=deletenodetest_coll_shard3_replica1 
url=http://127.0.0.1:43606/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 556240 INFO  (qtp22725655-3917) [    ] o.a.s.c.S.Request 
[deletenodetest_coll_shard2_replica2]  webapp=/solr path=/get 
params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2}
 status=0 QTime=1
   [junit4]   2> 556240 INFO  (qtp25187729-3717) [    ] o.a.s.c.SyncStrategy 
Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 556240 INFO  (qtp25187729-3717) [    ] 
o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we 
can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 556241 INFO  (qtp12370062-3722) [    ] o.a.s.c.S.Request 
[deletenodetest_coll_shard3_replica2]  webapp=/solr path=/get 
params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2}
 status=0 QTime=0
   [junit4]   2> 556242 INFO  (qtp20049145-3764) [    ] o.a.s.c.SyncStrategy 
Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 556242 INFO  (qtp20049145-3764) [    ] 
o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we 
can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 556244 INFO  (qtp25187729-3717) [    ] 
o.a.s.c.ShardLeaderElectionContext I am the new leader: 
http://127.0.0.1:37968/solr/deletenodetest_coll_shard2_replica1/ shard2
   [junit4]   2> 556245 INFO  (qtp20049145-3764) [    ] 
o.a.s.c.ShardLeaderElectionContext I am the new leader: 
http://127.0.0.1:43606/solr/deletenodetest_coll_shard3_replica1/ shard3
   [junit4]   2> 556347 INFO  (zkCallback-777-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/deletenodetest_coll/state.json] for collection 
[deletenodetest_coll] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 556347 INFO  (zkCallback-769-thread-2) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/deletenodetest_coll/state.json] for collection 
[deletenodetest_coll] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 556347 INFO  (zkCallback-776-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/deletenodetest_coll/state.json] for collection 
[deletenodetest_coll] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 556347 INFO  (zkCallback-778-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/deletenodetest_coll/state.json] for collection 
[deletenodetest_coll] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 556347 INFO  (zkCallback-779-thread-3) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/deletenodetest_coll/state.json] for collection 
[deletenodetest_coll] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 556348 INFO  (zkCallback-780-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/deletenodetest_coll/state.json] for collection 
[deletenodetest_coll] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 556373 INFO  (qtp25187729-3744) [    ] o.a.s.c.ZkController I 
am the leader, no recovery necessary
   [junit4]   2> 556374 INFO  (qtp25187729-3744) [    ] o.a.s.s.HttpSolrCall 
[admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&collection.configName=conf1&name=deletenodetest_coll_shard5_replica1&action=CREATE&numShards=5&collection=deletenodetest_coll&shard=shard5&wt=javabin&version=2}
 status=0 QTime=1768
   [junit4]   2> 556395 INFO  (qtp25187729-3717) [    ] o.a.s.c.ZkController I 
am the leader, no recovery necessary
   [junit4]   2> 556395 INFO  (qtp20049145-3764) [    ] o.a.s.c.ZkController I 
am the leader, no recovery necessary
   [junit4]   2> 556396 INFO  (qtp25187729-3717) [    ] o.a.s.s.HttpSolrCall 
[admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&collection.configName=conf1&name=deletenodetest_coll_shard2_replica1&action=CREATE&numShards=5&collection=deletenodetest_coll&shard=shard2&wt=javabin&version=2}
 status=0 QTime=1789
   [junit4]   2> 556396 INFO  (qtp20049145-3764) [    ] o.a.s.s.HttpSolrCall 
[admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&collection.configName=conf1&name=deletenodetest_coll_shard3_replica1&action=CREATE&numShards=5&collection=deletenodetest_coll&shard=shard3&wt=javabin&version=2}
 status=0 QTime=1790
   [junit4]   2> 556499 INFO  (zkCallback-777-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/deletenodetest_coll/state.json] for collection 
[deletenodetest_coll] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 556499 INFO  (zkCallback-776-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/deletenodetest_coll/state.json] for collection 
[deletenodetest_coll] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 556499 INFO  (zkCallback-769-thread-2) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/deletenodetest_coll/state.json] for collection 
[deletenodetest_coll] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 556499 INFO  (zkCallback-778-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/deletenodetest_coll/state.json] for collection 
[deletenodetest_coll] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 556499 INFO  (zkCallback-780-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/deletenodetest_coll/state.json] for collection 
[deletenodetest_coll] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 556499 INFO  (zkCallback-779-thread-2) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/deletenodetest_coll/state.json] for collection 
[deletenodetest_coll] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 556721 INFO  (qtp13527231-3721) [    ] o.a.s.c.ZkController 
Core needs to recover:deletenodetest_coll_shard4_replica1
   [junit4]   2> 556721 INFO  (updateExecutor-767-thread-1) [    ] 
o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 556721 INFO  (recoveryExecutor-768-thread-1) [    ] 
o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
   [junit4]   2> 556722 INFO  (recoveryExecutor-768-thread-1) [    ] 
o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
   [junit4]   2> 556722 INFO  (recoveryExecutor-768-thread-1) [    ] 
o.a.s.c.RecoveryStrategy Begin buffering updates. 
core=[deletenodetest_coll_shard4_replica1]
   [junit4]   2> 556722 INFO  (recoveryExecutor-768-thread-1) [    ] 
o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, 
tlog=null}
   [junit4]   2> 556722 INFO  (recoveryExecutor-768-thread-1) [    ] 
o.a.s.c.RecoveryStrategy Publishing state of core 
[deletenodetest_coll_shard4_replica1] as recovering, leader is 
[http://127.0.0.1:42537/solr/deletenodetest_coll_shard4_replica2/] and I am 
[http://127.0.0.1:37150/solr/deletenodetest_coll_shard4_replica1/]
   [junit4]   2> 556722 INFO  (qtp13527231-3721) [    ] o.a.s.s.HttpSolrCall 
[admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&collection.configName=conf1&name=deletenodetest_coll_shard4_replica1&action=CREATE&numShards=5&collection=deletenodetest_coll&shard=shard4&wt=javabin&version=2}
 status=0 QTime=2117
   [junit4]   2> 556723 INFO  (recoveryExecutor-768-thread-1) [    ] 
o.a.s.c.RecoveryStrategy Sending prep recovery command to 
[http://127.0.0.1:42537/solr]; [WaitForState: 
action=PREPRECOVERY&core=deletenodetest_coll_shard4_replica2&nodeName=127.0.0.1:37150_solr&coreNodeName=core_node1&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
   [junit4]   2> 556724 INFO  (qtp25347926-3914) [    ] 
o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node1, state: 
recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
   [junit4]   2> 556725 INFO  (qtp25347926-3914) [    ] 
o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see 
deletenodetest_coll_shard4_replica2 (shard4 of deletenodetest_coll) have state: 
recovering
   [junit4]   2> 556725 INFO  (qtp25347926-3914) [    ] 
o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): 
collection=deletenodetest_coll, shard=shard4, 
thisCore=deletenodetest_coll_shard4_replica2, leaderDoesNotNeedRecovery=false, 
isLeader? true, live=true, checkLive=true, currentState=down, 
localState=active, nodeName=127.0.0.1:37150_solr, coreNodeName=core_node1, 
onlyIfActiveCheckResult=false, nodeProps: 
core_node1:{"core":"deletenodetest_coll_shard4_replica1","base_url":"http://127.0.0.1:37150/solr","node_name":"127.0.0.1:37150_solr","state":"down"}
   [junit4]   2> 556739 INFO  (qtp12370062-3729) [    ] o.a.s.c.ZkController 
Core needs to recover:deletenodetest_coll_shard3_replica2
   [junit4]   2> 556739 INFO  (qtp22725655-3760) [    ] o.a.s.c.ZkController 
Core needs to recover:deletenodetest_coll_shard2_replica2
   [junit4]   2> 556739 INFO  (updateExecutor-757-thread-1) [    ] 
o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 556739 INFO  (updateExecutor-765-thread-1) [    ] 
o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 556740 INFO  (recoveryExecutor-758-thread-1) [    ] 
o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
   [junit4]   2> 556740 INFO  (recoveryExecutor-758-thread-1) [    ] 
o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
   [junit4]   2> 556740 INFO  (recoveryExecutor-758-thread-1) [    ] 
o.a.s.c.RecoveryStrategy Begin buffering updates. 
core=[deletenodetest_coll_shard3_replica2]
   [junit4]   2> 556740 INFO  (recoveryExecutor-758-thread-1) [    ] 
o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, 
tlog=null}
   [junit4]   2> 556740 INFO  (recoveryExecutor-758-thread-1) [    ] 
o.a.s.c.RecoveryStrategy Publishing state of core 
[deletenodetest_coll_shard3_replica2] as recovering, leader is 
[http://127.0.0.1:43606/solr/deletenodetest_coll_shard3_replica1/] and I am 
[http://127.0.0.1:32944/solr/deletenodetest_coll_shard3_replica2/]
   [junit4]   2> 556741 INFO  (qtp22725655-3760) [    ] o.a.s.s.HttpSolrCall 
[admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&collection.configName=conf1&name=deletenodetest_coll_shard2_replica2&action=CREATE&numShards=5&collection=deletenodetest_coll&shard=shard2&wt=javabin&version=2}
 status=0 QTime=2135
   [junit4]   2> 556741 INFO  (recoveryExecutor-766-thread-1) [    ] 
o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
   [junit4]   2> 556741 INFO  (qtp12370062-3729) [    ] o.a.s.s.HttpSolrCall 
[admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&collection.configName=conf1&name=deletenodetest_coll_shard3_replica2&action=CREATE&numShards=5&collection=deletenodetest_coll&shard=shard3&wt=javabin&version=2}
 status=0 QTime=2130
   [junit4]   2> 556741 INFO  (recoveryExecutor-766-thread-1) [    ] 
o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
   [junit4]   2> 556742 INFO  (recoveryExecutor-766-thread-1) [    ] 
o.a.s.c.RecoveryStrategy Begin buffering updates. 
core=[deletenodetest_coll_shard2_replica2]
   [junit4]   2> 556742 INFO  (recoveryExecutor-766-thread-1) [    ] 
o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, 
tlog=null}
   [junit4]   2> 556742 INFO  (recoveryExecutor-766-thread-1) [    ] 
o.a.s.c.RecoveryStrategy Publishing state of core 
[deletenodetest_coll_shard2_replica2] as recovering, leader is 
[http://127.0.0.1:37968/solr/deletenodetest_coll_shard2_replica1/] and I am 
[http://127.0.0.1:44287/solr/deletenodetest_coll_shard2_replica2/]
   [junit4]   2> 556743 INFO  (recoveryExecutor-758-thread-1) [    ] 
o.a.s.c.RecoveryStrategy Sending prep recovery command to 
[http://127.0.0.1:43606/solr]; [WaitForState: 
action=PREPRECOVERY&core=deletenodetest_coll_shard3_replica1&nodeName=127.0.0.1:32944_solr&coreNodeName=core_node8&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
   [junit4]   2> 556743 INFO  (recoveryExecutor-766-thread-1) [    ] 
o.a.s.c.RecoveryStrategy Sending prep recovery command to 
[http://127.0.0.1:37968/solr]; [WaitForState: 
action=PREPRECOVERY&core=deletenodetest_coll_shard2_replica1&nodeName=127.0.0.1:44287_solr&coreNodeName=core_node9&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
   [junit4]   2> 556744 INFO  (qtp20049145-3766) [    ] 
o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node8, state: 
recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
   [junit4]   2> 556744 INFO  (qtp25187729-3753) [    ] 
o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node9, state: 
recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
   [junit4]   2> 556744 INFO  (qtp20049145-3766) [    ] 
o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see 
deletenodetest_coll_shard3_replica1 (shard3 of deletenodetest_coll) have state: 
recovering
   [junit4]   2> 556744 INFO  (qtp20049145-3766) [    ] 
o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): 
collection=deletenodetest_coll, shard=shard3, 
thisCore=deletenodetest_coll_shard3_replica1, leaderDoesNotNeedRecovery=false, 
isLeader? true, live=true, checkLive=true, currentState=down, 
localState=active, nodeName=127.0.0.1:32944_solr, coreNodeName=core_node8, 
onlyIfActiveCheckResult=false, nodeProps: 
core_node8:{"core":"deletenodetest_coll_shard3_replica2","base_url":"http://127.0.0.1:32944/solr","node_name":"127.0.0.1:32944_solr","state":"down"}
   [junit4]   2> 556745 INFO  (qtp25187729-3753) [    ] 
o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see 
deletenodetest_coll_shard2_replica1 (shard2 of deletenodetest_coll) have state: 
recovering
   [junit4]   2> 556745 INFO  (qtp25187729-3753) [    ] 
o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): 
collection=deletenodetest_coll, shard=shard2, 
thisCore=deletenodetest_coll_shard2_replica1, leaderDoesNotNeedRecovery=false, 
isLeader? true, live=true, checkLive=true, currentState=down, 
localState=active, nodeName=127.0.0.1:44287_solr, coreNodeName=core_node9, 
onlyIfActiveCheckResult=false, nodeProps: 
core_node9:{"core":"deletenodetest_coll_shard2_replica2","base_url":"http://127.0.0.1:44287/solr","node_name":"127.0.0.1:44287_solr","state":"down"}
   [junit4]   2> 556747 INFO  (qtp22725655-3736) [    ] o.a.s.c.ZkController 
Core needs to recover:deletenodetest_coll_shard5_replica2
   [junit4]   2> 556747 INFO  (qtp25347926-3720) [    ] o.a.s.c.ZkController 
Core needs to recover:deletenodetest_coll_shard1_replica2
   [junit4]   2> 556747 INFO  (updateExecutor-765-thread-2) [    ] 
o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 556747 INFO  (recoveryExecutor-766-thread-2) [    ] 
o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
   [junit4]   2> 556747 INFO  (recoveryExecutor-766-thread-2) [    ] 
o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
   [junit4]   2> 556747 INFO  (updateExecutor-759-thread-1) [    ] 
o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 556747 INFO  (recoveryExecutor-766-thread-2) [    ] 
o.a.s.c.RecoveryStrategy Begin buffering updates. 
core=[deletenodetest_coll_shard5_replica2]
   [junit4]   2> 556747 INFO  (recoveryExecutor-766-thread-2) [    ] 
o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, 
tlog=null}
   [junit4]   2> 556747 INFO  (recoveryExecutor-766-thread-2) [    ] 
o.a.s.c.RecoveryStrategy Publishing state of core 
[deletenodetest_coll_shard5_replica2] as recovering, leader is 
[http://127.0.0.1:37968/solr/deletenodetest_coll_shard5_replica1/] and I am 
[http://127.0.0.1:44287/solr/deletenodetest_coll_shard5_replica2/]
   [junit4]   2> 556747 INFO  (recoveryExecutor-760-thread-1) [    ] 
o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
   [junit4]   2> 556748 INFO  (recoveryExecutor-760-thread-1) [    ] 
o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
   [junit4]   2> 556748 INFO  (recoveryExecutor-760-thread-1) [    ] 
o.a.s.c.RecoveryStrategy Begin buffering updates. 
core=[deletenodetest_coll_shard1_replica2]
   [junit4]   2> 556748 INFO  (recoveryExecutor-760-thread-1) [    ] 
o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, 
tlog=null}
   [junit4]   2> 556748 INFO  (recoveryExecutor-760-thread-1) [    ] 
o.a.s.c.RecoveryStrategy Publishing state of core 
[deletenodetest_coll_shard1_replica2] as recovering, leader is 
[http://127.0.0.1:37150/solr/deletenodetest_coll_shard1_replica1/] and I am 
[http://127.0.0.1:42537/solr/deletenodetest_coll_shard1_replica2/]
   [junit4]   2> 556748 INFO  (qtp22725655-3736) [    ] o.a.s.s.HttpSolrCall 
[admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&collection.configName=conf1&name=deletenodetest_coll_shard5_replica2&action=CREATE&numShards=5&collection=deletenodetest_coll&shard=shard5&wt=javabin&version=2}
 status=0 QTime=2141
   [junit4]   2> 556749 INFO  (qtp25347926-3720) [    ] o.a.s.s.HttpSolrCall 
[admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&collection.configName=conf1&name=deletenodetest_coll_shard1_replica2&action=CREATE&numShards=5&collection=deletenodetest_coll&shard=shard1&wt=javabin&version=2}
 status=0 QTime=2138
   [junit4]   2> 556751 INFO  (recoveryExecutor-766-thread-2) [    ] 
o.a.s.c.RecoveryStrategy Sending prep recovery command to 
[http://127.0.0.1:37968/solr]; [WaitForState: 
action=PREPRECOVERY&core=deletenodetest_coll_shard5_replica1&nodeName=127.0.0.1:44287_solr&coreNodeName=core_node10&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
   [junit4]   2> 556751 INFO  (recoveryExecutor-760-thread-1) [    ] 
o.a.s.c.RecoveryStrategy Sending prep recovery command to 
[http://127.0.0.1:37150/solr]; [WaitForState: 
action=PREPRECOVERY&core=deletenodetest_coll_shard1_replica1&nodeName=127.0.0.1:42537_solr&coreNodeName=core_node7&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
   [junit4]   2> 556758 INFO  (qtp25187729-3739) [    ] 
o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node10, state: 
recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
   [junit4]   2> 556758 INFO  (qtp13527231-3907) [    ] 
o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node7, state: 
recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
   [junit4]   2> 556759 INFO  (qtp25187729-3739) [    ] 
o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see 
deletenodetest_coll_shard5_replica1 (shard5 of deletenodetest_coll) have state: 
recovering
   [junit4]   2> 556759 INFO  (qtp25187729-3739) [    ] 
o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): 
collection=deletenodetest_coll, shard=shard5, 
thisCore=deletenodetest_coll_shard5_replica1, leaderDoesNotNeedRecovery=false, 
isLeader? true, live=true, checkLive=true, currentState=down, 
localState=active, nodeName=127.0.0.1:44287_solr, coreNodeName=core_node10, 
onlyIfActiveCheckResult=false, nodeProps: 
core_node10:{"core":"deletenodetest_coll_shard5_replica2","base_url":"http://127.0.0.1:44287/solr","node_name":"127.0.0.1:44287_solr","state":"down"}
   [junit4]   2> 556759 INFO  (qtp25347926-3750) [    ] 
o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 
30 seconds. Check all shard replicas
   [junit4]   2> 556759 INFO  (qtp13527231-3907) [    ] 
o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see 
deletenodetest_coll_shard1_replica1 (shard1 of deletenodetest_coll) have state: 
recovering
   [junit4]   2> 55675

[...truncated too long message...]

eerCollectionConfigSetProcessor-96676649508012050-127.0.0.1:37968_solr-n_0000000001)
 [    ] o.a.s.c.OverseerTaskProcessor According to ZK I 
(id=96676649508012050-127.0.0.1:37968_solr-n_0000000001) am no longer a leader.
   [junit4]   2> 568286 WARN  (zkCallback-769-thread-2) [    ] 
o.a.s.c.SyncStrategy Closed, skipping sync up.
   [junit4]   2> 568286 INFO  (zkCallback-769-thread-2) [    ] 
o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we 
can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 568286 INFO  (zkCallback-769-thread-2) [    ] o.a.s.c.SolrCore 
[deletenodetest_coll_shard3_replica2]  CLOSING SolrCore 
org.apache.solr.core.SolrCore@aae308
   [junit4]   2> 568288 INFO  (jetty-launcher-749-thread-5) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@14be3e1{/solr,null,UNAVAILABLE}
   [junit4]   2> 568426 WARN  (zkCallback-779-thread-5) [    ] 
o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: 
[KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 568427 INFO  (jetty-launcher-749-thread-3) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@1b14cb3{/solr,null,UNAVAILABLE}
   [junit4]   2> 568436 WARN  (zkCallback-776-thread-1) [    ] 
o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: 
[KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 568436 WARN  (zkCallback-778-thread-4) [    ] 
o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: 
[KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 568436 WARN  (zkCallback-777-thread-2) [    ] 
o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: 
[KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 568436 INFO  (jetty-launcher-749-thread-2) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@633ca7{/solr,null,UNAVAILABLE}
   [junit4]   2> 568436 INFO  (jetty-launcher-749-thread-6) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@15d261c{/solr,null,UNAVAILABLE}
   [junit4]   2> 568436 INFO  (jetty-launcher-749-thread-4) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@1b44cb9{/solr,null,UNAVAILABLE}
   [junit4]   2> 568438 INFO  
(SUITE-DeleteNodeTest-seed#[8313F26FCB3FBD32]-worker) [    ] 
o.a.s.c.ZkTestServer connecting to 127.0.0.1:34578 34578
   [junit4]   2> 568441 INFO  (Thread-719) [    ] o.a.s.c.ZkTestServer 
connecting to 127.0.0.1:34578 34578
   [junit4]   2> 568442 WARN  (Thread-719) [    ] o.a.s.c.ZkTestServer Watch 
limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2>        7       /solr/aliases.json
   [junit4]   2>        7       /solr/clusterprops.json
   [junit4]   2>        6       /solr/security.json
   [junit4]   2>        6       /solr/configs/conf1
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2>        7       /solr/clusterstate.json
   [junit4]   2>        6       /solr/collections/deletenodetest_coll/state.json
   [junit4]   2>        2       
/solr/overseer_elect/election/96676649508012048-127.0.0.1:44287_solr-n_0000000002
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2>        11      /solr/overseer/queue
   [junit4]   2>        9       /solr/overseer/collection-queue-work
   [junit4]   2>        7       /solr/live_nodes
   [junit4]   2>        7       /solr/collections
   [junit4]   2>        2       /solr/overseer/queue-work
   [junit4]   2> 
   [junit4]   2> 568442 INFO  
(SUITE-DeleteNodeTest-seed#[8313F26FCB3FBD32]-worker) [    ] 
o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> Sep 29, 2016 5:03:36 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 2 leaked 
thread(s).
   [junit4]   2> Sep 29, 2016 5:03:56 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at 
org.apache.solr.cloud.DeleteNodeTest: 
   [junit4]   2>    1) Thread[id=3972, 
name=OverseerHdfsCoreFailoverThread-96676649508012050-127.0.0.1:37968_solr-n_0000000001,
 state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
   [junit4]   2>         at java.lang.Thread.sleep(java.base@9-ea/Native Method)
   [junit4]   2>         at 
org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)
   [junit4]   2>         at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
   [junit4]   2> Sep 29, 2016 5:03:56 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=3972, 
name=OverseerHdfsCoreFailoverThread-96676649508012050-127.0.0.1:37968_solr-n_0000000001,
 state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
   [junit4]   2> Sep 29, 2016 5:03:59 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> SEVERE: There are still zombie threads that couldn't be 
terminated:
   [junit4]   2>    1) Thread[id=3972, 
name=OverseerHdfsCoreFailoverThread-96676649508012050-127.0.0.1:37968_solr-n_0000000001,
 state=RUNNABLE, group=Overseer Hdfs SolrCore Failover Thread.]
   [junit4]   2>         at 
java.lang.Throwable.fillInStackTrace(java.base@9-ea/Native Method)
   [junit4]   2>         at 
java.lang.Throwable.fillInStackTrace(java.base@9-ea/Throwable.java:788)
   [junit4]   2>         at 
java.lang.Throwable.<init>(java.base@9-ea/Throwable.java:271)
   [junit4]   2>         at 
java.lang.Exception.<init>(java.base@9-ea/Exception.java:66)
   [junit4]   2>         at 
java.lang.InterruptedException.<init>(java.base@9-ea/InterruptedException.java:67)
   [junit4]   2>         at java.lang.Thread.sleep(java.base@9-ea/Native Method)
   [junit4]   2>         at 
org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)
   [junit4]   2>         at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62): {}, 
docValues:{}, maxPointsInLeafNode=497, maxMBSortInHeap=6.094387221985047, 
sim=RandomSimilarity(queryNorm=true): {}, locale=en-HK, timezone=Mexico/General
   [junit4]   2> NOTE: Linux 4.4.0-36-generic i386/Oracle Corporation 9-ea 
(32-bit)/cpus=12,threads=2,free=156881144,total=272105472
   [junit4]   2> NOTE: All tests run in this JVM: [TestSolrFieldCacheMBean, 
TestUseDocValuesAsStored, AddBlockUpdateTest, CloudMLTQParserTest, 
TestHighFrequencyDictionaryFactory, ForceLeaderTest, BasicAuthIntegrationTest, 
TestRawTransformer, TestUniqueKeyFieldResource, RollingRestartTest, 
TestGeoJSONResponseWriter, ExplicitHLLTest, RequestHandlersTest, 
BlockJoinFacetRandomTest, HLLUtilTest, CoreAdminHandlerTest, AssignTest, 
ExitableDirectoryReaderTest, TestGroupingSearch, SolrXmlInZkTest, 
SpatialRPTFieldTypeTest, TestSort, TestFieldCacheVsDocValues, UpdateParamsTest, 
TestIndexingPerformance, TestFastOutputStream, DebugComponentTest, 
OverseerRolesTest, OverseerTest, DocExpirationUpdateProcessorFactoryTest, 
TestExactSharedStatsCache, TestSortByMinMaxFunction, ResponseHeaderTest, 
TestPKIAuthenticationPlugin, TestDocTermOrds, SolrTestCaseJ4Test, 
LeaderFailureAfterFreshStartTest, TestMacroExpander, TestSolrJ, 
TestReplicaProperties, TestNumericTerms32, TestStressReorder, 
TestDownShardTolerantSearch, DocValuesMultiTest, TestConfig, ZkStateWriterTest, 
TestRandomCollapseQParserPlugin, DirectUpdateHandlerOptimizeTest, 
TestBinaryField, TestLMDirichletSimilarityFactory, AutoCommitTest, 
DeleteNodeTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=DeleteNodeTest 
-Dtests.seed=8313F26FCB3FBD32 -Dtests.multiplier=3 -Dtests.slow=true 
-Dtests.locale=en-HK -Dtests.timezone=Mexico/General -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J2 | DeleteNodeTest (suite) <<<
   [junit4]    > Throwable #1: 
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE 
scope at org.apache.solr.cloud.DeleteNodeTest: 
   [junit4]    >    1) Thread[id=3972, 
name=OverseerHdfsCoreFailoverThread-96676649508012050-127.0.0.1:37968_solr-n_0000000001,
 state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
   [junit4]    >         at java.lang.Thread.sleep(java.base@9-ea/Native Method)
   [junit4]    >         at 
org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)
   [junit4]    >         at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([8313F26FCB3FBD32]:0)Throwable #2: 
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie 
threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=3972, 
name=OverseerHdfsCoreFailoverThread-96676649508012050-127.0.0.1:37968_solr-n_0000000001,
 state=RUNNABLE, group=Overseer Hdfs SolrCore Failover Thread.]
   [junit4]    >         at 
java.lang.Throwable.fillInStackTrace(java.base@9-ea/Native Method)
   [junit4]    >         at 
java.lang.Throwable.fillInStackTrace(java.base@9-ea/Throwable.java:788)
   [junit4]    >         at 
java.lang.Throwable.<init>(java.base@9-ea/Throwable.java:271)
   [junit4]    >         at 
java.lang.Exception.<init>(java.base@9-ea/Exception.java:66)
   [junit4]    >         at 
java.lang.InterruptedException.<init>(java.base@9-ea/InterruptedException.java:67)
   [junit4]    >         at java.lang.Thread.sleep(java.base@9-ea/Native Method)
   [junit4]    >         at 
org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)
   [junit4]    >         at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([8313F26FCB3FBD32]:0)
   [junit4] Completed [170/634 (1!)] on J2 in 37.28s, 1 test, 2 errors <<< 
FAILURES!

[...truncated 51268 lines...]

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to