Build: https://builds.apache.org/job/Lucene-Solr-Tests-6.x/465/

1 tests failed.
FAILED:  
junit.framework.TestSuite.org.apache.solr.response.transform.TestSubQueryTransformerDistrib

Error Message:
1 thread leaked from SUITE scope at 
org.apache.solr.response.transform.TestSubQueryTransformerDistrib:     1) 
Thread[id=1369, 
name=OverseerHdfsCoreFailoverThread-96706052996988937-127.0.0.1:38475_solr-n_0000000001,
 state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)
         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE 
scope at org.apache.solr.response.transform.TestSubQueryTransformerDistrib: 
   1) Thread[id=1369, 
name=OverseerHdfsCoreFailoverThread-96706052996988937-127.0.0.1:38475_solr-n_0000000001,
 state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)
        at java.lang.Thread.run(Thread.java:745)
        at __randomizedtesting.SeedInfo.seed([B6C73D6CCF57C41A]:0)




Build Log:
[...truncated 10791 lines...]
   [junit4] Suite: 
org.apache.solr.response.transform.TestSubQueryTransformerDistrib
   [junit4]   2> Creating dataDir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/init-core-data-001
   [junit4]   2> 347909 INFO  
(SUITE-TestSubQueryTransformerDistrib-seed#[B6C73D6CCF57C41A]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: 
@org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=None)
   [junit4]   2> 347910 INFO  
(SUITE-TestSubQueryTransformerDistrib-seed#[B6C73D6CCF57C41A]-worker) [    ] 
o.a.s.c.MiniSolrCloudCluster Starting cluster of 5 servers in 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001
   [junit4]   2> 347910 INFO  
(SUITE-TestSubQueryTransformerDistrib-seed#[B6C73D6CCF57C41A]-worker) [    ] 
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 347913 INFO  (Thread-392) [    ] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 347914 INFO  (Thread-392) [    ] o.a.s.c.ZkTestServer Starting 
server
   [junit4]   2> 348017 INFO  
(SUITE-TestSubQueryTransformerDistrib-seed#[B6C73D6CCF57C41A]-worker) [    ] 
o.a.s.c.ZkTestServer start zk server on port:51643
   [junit4]   2> 348033 INFO  (jetty-launcher-151-thread-1) [    ] 
o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 348038 INFO  (jetty-launcher-151-thread-5) [    ] 
o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 348039 INFO  (jetty-launcher-151-thread-4) [    ] 
o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 348047 INFO  (jetty-launcher-151-thread-2) [    ] 
o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 348098 INFO  (jetty-launcher-151-thread-5) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@22222ad1{/solr,null,AVAILABLE}
   [junit4]   2> 348099 INFO  (jetty-launcher-151-thread-5) [    ] 
o.e.j.s.ServerConnector Started 
ServerConnector@4d016fb0{HTTP/1.1,[http/1.1]}{127.0.0.1:39449}
   [junit4]   2> 348099 INFO  (jetty-launcher-151-thread-5) [    ] 
o.e.j.s.Server Started @356564ms
   [junit4]   2> 348099 INFO  (jetty-launcher-151-thread-5) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=39449}
   [junit4]   2> 348099 INFO  (jetty-launcher-151-thread-5) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 
6.3.0
   [junit4]   2> 348099 INFO  (jetty-launcher-151-thread-5) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 348099 INFO  (jetty-launcher-151-thread-5) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 348099 INFO  (jetty-launcher-151-thread-5) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2016-10-04T21:41:03.932Z
   [junit4]   2> 348222 INFO  (jetty-launcher-151-thread-1) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@2e50d386{/solr,null,AVAILABLE}
   [junit4]   2> 348222 INFO  (jetty-launcher-151-thread-1) [    ] 
o.e.j.s.ServerConnector Started 
ServerConnector@74d6604d{HTTP/1.1,[http/1.1]}{127.0.0.1:38475}
   [junit4]   2> 348222 INFO  (jetty-launcher-151-thread-1) [    ] 
o.e.j.s.Server Started @356688ms
   [junit4]   2> 348222 INFO  (jetty-launcher-151-thread-1) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=38475}
   [junit4]   2> 348223 INFO  (jetty-launcher-151-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 
6.3.0
   [junit4]   2> 348223 INFO  (jetty-launcher-151-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 348223 INFO  (jetty-launcher-151-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 348223 INFO  (jetty-launcher-151-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2016-10-04T21:41:04.056Z
   [junit4]   2> 348228 INFO  (jetty-launcher-151-thread-3) [    ] 
o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 348252 INFO  (jetty-launcher-151-thread-2) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@4cd97e8e{/solr,null,AVAILABLE}
   [junit4]   2> 348253 INFO  (jetty-launcher-151-thread-2) [    ] 
o.e.j.s.ServerConnector Started 
ServerConnector@f5426ec{HTTP/1.1,[http/1.1]}{127.0.0.1:54784}
   [junit4]   2> 348253 INFO  (jetty-launcher-151-thread-2) [    ] 
o.e.j.s.Server Started @356718ms
   [junit4]   2> 348253 INFO  (jetty-launcher-151-thread-2) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=54784}
   [junit4]   2> 348253 INFO  (jetty-launcher-151-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 
6.3.0
   [junit4]   2> 348253 INFO  (jetty-launcher-151-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 348253 INFO  (jetty-launcher-151-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 348253 INFO  (jetty-launcher-151-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2016-10-04T21:41:04.086Z
   [junit4]   2> 348285 INFO  (jetty-launcher-151-thread-4) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@75f1e9bd{/solr,null,AVAILABLE}
   [junit4]   2> 348285 INFO  (jetty-launcher-151-thread-4) [    ] 
o.e.j.s.ServerConnector Started 
ServerConnector@1a7a90c7{HTTP/1.1,[http/1.1]}{127.0.0.1:58449}
   [junit4]   2> 348285 INFO  (jetty-launcher-151-thread-4) [    ] 
o.e.j.s.Server Started @356751ms
   [junit4]   2> 348285 INFO  (jetty-launcher-151-thread-4) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=58449}
   [junit4]   2> 348286 INFO  (jetty-launcher-151-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 
6.3.0
   [junit4]   2> 348286 INFO  (jetty-launcher-151-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 348286 INFO  (jetty-launcher-151-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 348286 INFO  (jetty-launcher-151-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2016-10-04T21:41:04.119Z
   [junit4]   2> 348338 INFO  (jetty-launcher-151-thread-3) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@6a6770cc{/solr,null,AVAILABLE}
   [junit4]   2> 348339 INFO  (jetty-launcher-151-thread-3) [    ] 
o.e.j.s.ServerConnector Started 
ServerConnector@6b5c71b1{HTTP/1.1,[http/1.1]}{127.0.0.1:34130}
   [junit4]   2> 348339 INFO  (jetty-launcher-151-thread-3) [    ] 
o.e.j.s.Server Started @356805ms
   [junit4]   2> 348339 INFO  (jetty-launcher-151-thread-3) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=34130}
   [junit4]   2> 348339 INFO  (jetty-launcher-151-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 
6.3.0
   [junit4]   2> 348339 INFO  (jetty-launcher-151-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 348339 INFO  (jetty-launcher-151-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 348339 INFO  (jetty-launcher-151-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2016-10-04T21:41:04.172Z
   [junit4]   2> 348378 INFO  (jetty-launcher-151-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 348391 INFO  (jetty-launcher-151-thread-2) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 348395 INFO  (jetty-launcher-151-thread-2) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51643/solr
   [junit4]   2> 348426 INFO  (jetty-launcher-151-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 348496 INFO  (jetty-launcher-151-thread-5) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 348494 INFO  (jetty-launcher-151-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 348434 INFO  (jetty-launcher-151-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 348626 INFO  (jetty-launcher-151-thread-5) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 348626 INFO  (jetty-launcher-151-thread-5) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51643/solr
   [junit4]   2> 348630 INFO  (jetty-launcher-151-thread-4) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 348646 INFO  (jetty-launcher-151-thread-1) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 348649 INFO  (jetty-launcher-151-thread-1) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51643/solr
   [junit4]   2> 348663 INFO  (jetty-launcher-151-thread-2) 
[n:127.0.0.1:54784_solr    ] o.a.s.c.OverseerElectionContext I am going to be 
the leader 127.0.0.1:54784_solr
   [junit4]   2> 348688 INFO  (jetty-launcher-151-thread-2) 
[n:127.0.0.1:54784_solr    ] o.a.s.c.Overseer Overseer 
(id=96706052996988935-127.0.0.1:54784_solr-n_0000000000) starting
   [junit4]   2> 348694 INFO  (jetty-launcher-151-thread-1) 
[n:127.0.0.1:38475_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:38475_solr
   [junit4]   2> 348696 INFO  (jetty-launcher-151-thread-4) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51643/solr
   [junit4]   2> 348759 INFO  (jetty-launcher-151-thread-3) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 348794 INFO  
(zkCallback-161-thread-1-processing-n:127.0.0.1:54784_solr) 
[n:127.0.0.1:54784_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 348896 INFO  
(zkCallback-171-thread-1-processing-n:127.0.0.1:38475_solr) 
[n:127.0.0.1:38475_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 348900 INFO  (jetty-launcher-151-thread-1) 
[n:127.0.0.1:38475_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node1/.
   [junit4]   2> 348901 INFO  (jetty-launcher-151-thread-3) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51643/solr
   [junit4]   2> 349199 INFO  (jetty-launcher-151-thread-2) 
[n:127.0.0.1:54784_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:54784_solr
   [junit4]   2> 349257 INFO  (jetty-launcher-151-thread-5) 
[n:127.0.0.1:39449_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 349554 INFO  
(zkCallback-171-thread-1-processing-n:127.0.0.1:38475_solr) 
[n:127.0.0.1:38475_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (1) -> (2)
   [junit4]   2> 349555 INFO  
(zkCallback-161-thread-1-processing-n:127.0.0.1:54784_solr) 
[n:127.0.0.1:54784_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (1) -> (2)
   [junit4]   2> 349592 INFO  (jetty-launcher-151-thread-3) 
[n:127.0.0.1:34130_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (2)
   [junit4]   2> 349598 INFO  (jetty-launcher-151-thread-5) 
[n:127.0.0.1:39449_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:39449_solr
   [junit4]   2> 349621 INFO  
(zkCallback-171-thread-1-processing-n:127.0.0.1:38475_solr) 
[n:127.0.0.1:38475_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (2) -> (3)
   [junit4]   2> 349622 INFO  (jetty-launcher-151-thread-4) 
[n:127.0.0.1:58449_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (3)
   [junit4]   2> 349623 INFO  (jetty-launcher-151-thread-3) 
[n:127.0.0.1:34130_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:34130_solr
   [junit4]   2> 349625 INFO  
(zkCallback-161-thread-1-processing-n:127.0.0.1:54784_solr) 
[n:127.0.0.1:54784_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (2) -> (3)
   [junit4]   2> 349627 INFO  
(zkCallback-177-thread-1-processing-n:127.0.0.1:39449_solr) 
[n:127.0.0.1:39449_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (1) -> (2)
   [junit4]   2> 349662 INFO  
(zkCallback-177-thread-2-processing-n:127.0.0.1:39449_solr) 
[n:127.0.0.1:39449_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (2) -> (4)
   [junit4]   2> 349667 INFO  (jetty-launcher-151-thread-2) 
[n:127.0.0.1:54784_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node2/.
   [junit4]   2> 349672 INFO  (jetty-launcher-151-thread-4) 
[n:127.0.0.1:58449_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:58449_solr
   [junit4]   2> 349673 INFO  
(zkCallback-171-thread-1-processing-n:127.0.0.1:38475_solr) 
[n:127.0.0.1:38475_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (3) -> (4)
   [junit4]   2> 349680 INFO  (jetty-launcher-151-thread-5) 
[n:127.0.0.1:39449_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node3/.
   [junit4]   2> 349684 INFO  (jetty-launcher-151-thread-3) 
[n:127.0.0.1:34130_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node5/.
   [junit4]   2> 349686 INFO  
(zkCallback-171-thread-1-processing-n:127.0.0.1:38475_solr) 
[n:127.0.0.1:38475_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (4) -> (5)
   [junit4]   2> 349689 INFO  
(zkCallback-177-thread-2-processing-n:127.0.0.1:39449_solr) 
[n:127.0.0.1:39449_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (4) -> (5)
   [junit4]   2> 349690 INFO  
(zkCallback-161-thread-2-processing-n:127.0.0.1:54784_solr) 
[n:127.0.0.1:54784_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (3) -> (5)
   [junit4]   2> 349734 INFO  
(zkCallback-180-thread-1-processing-n:127.0.0.1:58449_solr) 
[n:127.0.0.1:58449_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (3) -> (5)
   [junit4]   2> 349734 INFO  
(zkCallback-178-thread-1-processing-n:127.0.0.1:34130_solr) 
[n:127.0.0.1:34130_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (2) -> (5)
   [junit4]   2> 349791 INFO  (jetty-launcher-151-thread-4) 
[n:127.0.0.1:58449_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node4/.
   [junit4]   2> 350011 INFO  
(SUITE-TestSubQueryTransformerDistrib-seed#[B6C73D6CCF57C41A]-worker) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (5)
   [junit4]   2> 352073 INFO  (qtp557959012-1153) [n:127.0.0.1:39449_solr    ] 
o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params 
replicationFactor=2&property.schema=schema-docValuesJoin.xml&property.config=solrconfig-doctransformers.xml&collection.configName=solrCloudCollectionConfig&name=people&action=CREATE&numShards=2&wt=javabin&version=2
 and sendToOCPQueue=true
   [junit4]   2> 352087 INFO  
(OverseerThreadFactory-448-thread-1-processing-n:127.0.0.1:54784_solr) 
[n:127.0.0.1:54784_solr    ] o.a.s.c.CreateCollectionCmd Create collection 
people
   [junit4]   2> 352263 INFO  (qtp557959012-1151) [n:127.0.0.1:39449_solr    ] 
o.a.s.h.a.CoreAdminOperation core create command 
property.schema=schema-docValuesJoin.xml&property.config=solrconfig-doctransformers.xml&qt=/admin/cores&collection.configName=solrCloudCollectionConfig&name=people_shard2_replica2&action=CREATE&numShards=2&collection=people&shard=shard2&wt=javabin&version=2
   [junit4]   2> 352290 INFO  (qtp1201973810-1191) [n:127.0.0.1:58449_solr    ] 
o.a.s.h.a.CoreAdminOperation core create command 
property.schema=schema-docValuesJoin.xml&property.config=solrconfig-doctransformers.xml&qt=/admin/cores&collection.configName=solrCloudCollectionConfig&name=people_shard1_replica1&action=CREATE&numShards=2&collection=people&shard=shard1&wt=javabin&version=2
   [junit4]   2> 352291 INFO  (qtp1231523479-1176) [n:127.0.0.1:54784_solr    ] 
o.a.s.h.a.CoreAdminOperation core create command 
property.schema=schema-docValuesJoin.xml&property.config=solrconfig-doctransformers.xml&qt=/admin/cores&collection.configName=solrCloudCollectionConfig&name=people_shard2_replica1&action=CREATE&numShards=2&collection=people&shard=shard2&wt=javabin&version=2
   [junit4]   2> 352292 INFO  (qtp1663009772-1163) [n:127.0.0.1:38475_solr    ] 
o.a.s.h.a.CoreAdminOperation core create command 
property.schema=schema-docValuesJoin.xml&property.config=solrconfig-doctransformers.xml&qt=/admin/cores&collection.configName=solrCloudCollectionConfig&name=people_shard1_replica2&action=CREATE&numShards=2&collection=people&shard=shard1&wt=javabin&version=2
   [junit4]   2> 352413 INFO  
(zkCallback-180-thread-1-processing-n:127.0.0.1:58449_solr) 
[n:127.0.0.1:58449_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/people/state.json] for collection [people] has occurred - 
updating... (live nodes size: [5])
   [junit4]   2> 352413 INFO  
(zkCallback-171-thread-1-processing-n:127.0.0.1:38475_solr) 
[n:127.0.0.1:38475_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/people/state.json] for collection [people] has occurred - 
updating... (live nodes size: [5])
   [junit4]   2> 352413 INFO  
(zkCallback-177-thread-2-processing-n:127.0.0.1:39449_solr) 
[n:127.0.0.1:39449_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/people/state.json] for collection [people] has occurred - 
updating... (live nodes size: [5])
   [junit4]   2> 352413 INFO  
(zkCallback-161-thread-1-processing-n:127.0.0.1:54784_solr) 
[n:127.0.0.1:54784_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/people/state.json] for collection [people] has occurred - 
updating... (live nodes size: [5])
   [junit4]   2> 353345 INFO  (qtp1663009772-1163) [n:127.0.0.1:38475_solr 
c:people s:shard1  x:people_shard1_replica2] o.a.s.c.SolrConfig Using Lucene 
MatchVersion: 6.3.0
   [junit4]   2> 353348 INFO  (qtp1201973810-1191) [n:127.0.0.1:58449_solr 
c:people s:shard1  x:people_shard1_replica1] o.a.s.c.SolrConfig Using Lucene 
MatchVersion: 6.3.0
   [junit4]   2> 353355 INFO  (qtp1231523479-1176) [n:127.0.0.1:54784_solr 
c:people s:shard2  x:people_shard2_replica1] o.a.s.c.SolrConfig Using Lucene 
MatchVersion: 6.3.0
   [junit4]   2> 353362 INFO  (qtp557959012-1151) [n:127.0.0.1:39449_solr 
c:people s:shard2  x:people_shard2_replica2] o.a.s.c.SolrConfig Using Lucene 
MatchVersion: 6.3.0
   [junit4]   2> 353376 INFO  (qtp1663009772-1163) [n:127.0.0.1:38475_solr 
c:people s:shard1  x:people_shard1_replica2] o.a.s.s.IndexSchema 
[people_shard1_replica2] Schema name=doc-values-for-Join
   [junit4]   2> 353398 INFO  (qtp1201973810-1191) [n:127.0.0.1:58449_solr 
c:people s:shard1  x:people_shard1_replica1] o.a.s.s.IndexSchema 
[people_shard1_replica1] Schema name=doc-values-for-Join
   [junit4]   2> 353472 WARN  (qtp1663009772-1163) [n:127.0.0.1:38475_solr 
c:people s:shard1  x:people_shard1_replica2] o.a.s.s.IndexSchema 
[people_shard1_replica2] default search field in schema is id. WARNING: 
Deprecated, please use 'df' on request instead.
   [junit4]   2> 353473 INFO  (qtp1663009772-1163) [n:127.0.0.1:38475_solr 
c:people s:shard1  x:people_shard1_replica2] o.a.s.s.IndexSchema Loaded schema 
doc-values-for-Join/1.6 with uniqueid field id
   [junit4]   2> 353480 INFO  (qtp557959012-1151) [n:127.0.0.1:39449_solr 
c:people s:shard2  x:people_shard2_replica2] o.a.s.s.IndexSchema 
[people_shard2_replica2] Schema name=doc-values-for-Join
   [junit4]   2> 353483 INFO  (qtp1231523479-1176) [n:127.0.0.1:54784_solr 
c:people s:shard2  x:people_shard2_replica1] o.a.s.s.IndexSchema 
[people_shard2_replica1] Schema name=doc-values-for-Join
   [junit4]   2> 353497 WARN  (qtp1201973810-1191) [n:127.0.0.1:58449_solr 
c:people s:shard1  x:people_shard1_replica1] o.a.s.s.IndexSchema 
[people_shard1_replica1] default search field in schema is id. WARNING: 
Deprecated, please use 'df' on request instead.
   [junit4]   2> 353531 INFO  (qtp1201973810-1191) [n:127.0.0.1:58449_solr 
c:people s:shard1  x:people_shard1_replica1] o.a.s.s.IndexSchema Loaded schema 
doc-values-for-Join/1.6 with uniqueid field id
   [junit4]   2> 353530 WARN  (qtp1231523479-1176) [n:127.0.0.1:54784_solr 
c:people s:shard2  x:people_shard2_replica1] o.a.s.s.IndexSchema 
[people_shard2_replica1] default search field in schema is id. WARNING: 
Deprecated, please use 'df' on request instead.
   [junit4]   2> 353532 INFO  (qtp1663009772-1163) [n:127.0.0.1:38475_solr 
c:people s:shard1  x:people_shard1_replica2] o.a.s.c.CoreContainer Creating 
SolrCore 'people_shard1_replica2' using configuration from collection people
   [junit4]   2> 353533 INFO  (qtp1663009772-1163) [n:127.0.0.1:38475_solr 
c:people s:shard1 r:core_node2 x:people_shard1_replica2] o.a.s.c.SolrCore 
[[people_shard1_replica2] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node1/people_shard1_replica2],
 
dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node1/./people_shard1_replica2/data/]
   [junit4]   2> 353500 WARN  (qtp557959012-1151) [n:127.0.0.1:39449_solr 
c:people s:shard2  x:people_shard2_replica2] o.a.s.s.IndexSchema 
[people_shard2_replica2] default search field in schema is id. WARNING: 
Deprecated, please use 'df' on request instead.
   [junit4]   2> 353538 INFO  (qtp1201973810-1191) [n:127.0.0.1:58449_solr 
c:people s:shard1  x:people_shard1_replica1] o.a.s.c.CoreContainer Creating 
SolrCore 'people_shard1_replica1' using configuration from collection people
   [junit4]   2> 353538 INFO  (qtp1201973810-1191) [n:127.0.0.1:58449_solr 
c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.c.SolrCore 
[[people_shard1_replica1] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node4/people_shard1_replica1],
 
dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node4/./people_shard1_replica1/data/]
   [junit4]   2> 353540 INFO  (qtp1231523479-1176) [n:127.0.0.1:54784_solr 
c:people s:shard2  x:people_shard2_replica1] o.a.s.s.IndexSchema Loaded schema 
doc-values-for-Join/1.6 with uniqueid field id
   [junit4]   2> 353540 INFO  (qtp557959012-1151) [n:127.0.0.1:39449_solr 
c:people s:shard2  x:people_shard2_replica2] o.a.s.s.IndexSchema Loaded schema 
doc-values-for-Join/1.6 with uniqueid field id
   [junit4]   2> 353547 INFO  (qtp557959012-1151) [n:127.0.0.1:39449_solr 
c:people s:shard2  x:people_shard2_replica2] o.a.s.c.CoreContainer Creating 
SolrCore 'people_shard2_replica2' using configuration from collection people
   [junit4]   2> 353548 INFO  (qtp557959012-1151) [n:127.0.0.1:39449_solr 
c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.c.SolrCore 
[[people_shard2_replica2] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node3/people_shard2_replica2],
 
dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node3/./people_shard2_replica2/data/]
   [junit4]   2> 353549 INFO  (qtp1231523479-1176) [n:127.0.0.1:54784_solr 
c:people s:shard2  x:people_shard2_replica1] o.a.s.c.CoreContainer Creating 
SolrCore 'people_shard2_replica1' using configuration from collection people
   [junit4]   2> 353549 INFO  (qtp1231523479-1176) [n:127.0.0.1:54784_solr 
c:people s:shard2 r:core_node4 x:people_shard2_replica1] o.a.s.c.SolrCore 
[[people_shard2_replica1] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node2/people_shard2_replica1],
 
dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node2/./people_shard2_replica1/data/]
   [junit4]   2> 353694 INFO  (qtp1663009772-1163) [n:127.0.0.1:38475_solr 
c:people s:shard1 r:core_node2 x:people_shard1_replica2] o.a.s.u.UpdateHandler 
Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 353695 INFO  (qtp1663009772-1163) [n:127.0.0.1:38475_solr 
c:people s:shard1 r:core_node2 x:people_shard1_replica2] o.a.s.u.UpdateLog 
Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 
maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 353696 INFO  (qtp1663009772-1163) [n:127.0.0.1:38475_solr 
c:people s:shard1 r:core_node2 x:people_shard1_replica2] o.a.s.u.CommitTracker 
Hard AutoCommit: disabled
   [junit4]   2> 353696 INFO  (qtp1663009772-1163) [n:127.0.0.1:38475_solr 
c:people s:shard1 r:core_node2 x:people_shard1_replica2] o.a.s.u.CommitTracker 
Soft AutoCommit: disabled
   [junit4]   2> 353697 INFO  (qtp1663009772-1163) [n:127.0.0.1:38475_solr 
c:people s:shard1 r:core_node2 x:people_shard1_replica2] 
o.a.s.s.SolrIndexSearcher Opening [Searcher@3a47d050[people_shard1_replica2] 
main]
   [junit4]   2> 353699 INFO  (qtp1663009772-1163) [n:127.0.0.1:38475_solr 
c:people s:shard1 r:core_node2 x:people_shard1_replica2] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/solrCloudCollectionConfig
   [junit4]   2> 353699 INFO  (qtp1663009772-1163) [n:127.0.0.1:38475_solr 
c:people s:shard1 r:core_node2 x:people_shard1_replica2] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/solrCloudCollectionConfig
   [junit4]   2> 353699 INFO  (qtp1663009772-1163) [n:127.0.0.1:38475_solr 
c:people s:shard1 r:core_node2 x:people_shard1_replica2] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 353737 INFO  (qtp1231523479-1176) [n:127.0.0.1:54784_solr 
c:people s:shard2 r:core_node4 x:people_shard2_replica1] o.a.s.u.UpdateHandler 
Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 353738 INFO  (qtp1231523479-1176) [n:127.0.0.1:54784_solr 
c:people s:shard2 r:core_node4 x:people_shard2_replica1] o.a.s.u.UpdateLog 
Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 
maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 353738 INFO  (qtp1231523479-1176) [n:127.0.0.1:54784_solr 
c:people s:shard2 r:core_node4 x:people_shard2_replica1] o.a.s.u.CommitTracker 
Hard AutoCommit: disabled
   [junit4]   2> 353739 INFO  (qtp1231523479-1176) [n:127.0.0.1:54784_solr 
c:people s:shard2 r:core_node4 x:people_shard2_replica1] o.a.s.u.CommitTracker 
Soft AutoCommit: disabled
   [junit4]   2> 353739 INFO  (qtp1663009772-1163) [n:127.0.0.1:38475_solr 
c:people s:shard1 r:core_node2 x:people_shard1_replica2] o.a.s.u.UpdateLog 
Could not find max version in index or recent updates, using new clock 
1547296854058729472
   [junit4]   2> 353741 INFO  
(searcherExecutor-457-thread-1-processing-n:127.0.0.1:38475_solr 
x:people_shard1_replica2 s:shard1 c:people r:core_node2) 
[n:127.0.0.1:38475_solr c:people s:shard1 r:core_node2 
x:people_shard1_replica2] o.a.s.c.SolrCore [people_shard1_replica2] Registered 
new searcher Searcher@3a47d050[people_shard1_replica2] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 353753 INFO  (qtp1663009772-1163) [n:127.0.0.1:38475_solr 
c:people s:shard1 r:core_node2 x:people_shard1_replica2] 
o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for 
shard shard1: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 353766 INFO  (qtp1201973810-1191) [n:127.0.0.1:58449_solr 
c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.u.UpdateHandler 
Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 353766 INFO  (qtp1201973810-1191) [n:127.0.0.1:58449_solr 
c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.u.UpdateLog 
Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 
maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 353767 INFO  (qtp1201973810-1191) [n:127.0.0.1:58449_solr 
c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.u.CommitTracker 
Hard AutoCommit: disabled
   [junit4]   2> 353767 INFO  (qtp1201973810-1191) [n:127.0.0.1:58449_solr 
c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.u.CommitTracker 
Soft AutoCommit: disabled
   [junit4]   2> 353767 INFO  (qtp1201973810-1191) [n:127.0.0.1:58449_solr 
c:people s:shard1 r:core_node3 x:people_shard1_replica1] 
o.a.s.s.SolrIndexSearcher Opening [Searcher@b7bf2ee[people_shard1_replica1] 
main]
   [junit4]   2> 353769 INFO  (qtp1201973810-1191) [n:127.0.0.1:58449_solr 
c:people s:shard1 r:core_node3 x:people_shard1_replica1] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/solrCloudCollectionConfig
   [junit4]   2> 353769 INFO  (qtp1201973810-1191) [n:127.0.0.1:58449_solr 
c:people s:shard1 r:core_node3 x:people_shard1_replica1] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/solrCloudCollectionConfig
   [junit4]   2> 353769 INFO  (qtp1201973810-1191) [n:127.0.0.1:58449_solr 
c:people s:shard1 r:core_node3 x:people_shard1_replica1] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 353769 INFO  (qtp1201973810-1191) [n:127.0.0.1:58449_solr 
c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.u.UpdateLog 
Could not find max version in index or recent updates, using new clock 
1547296854090186752
   [junit4]   2> 353774 INFO  
(searcherExecutor-458-thread-1-processing-n:127.0.0.1:58449_solr 
x:people_shard1_replica1 s:shard1 c:people r:core_node3) 
[n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 
x:people_shard1_replica1] o.a.s.c.SolrCore [people_shard1_replica1] Registered 
new searcher Searcher@b7bf2ee[people_shard1_replica1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 353775 INFO  (qtp1231523479-1176) [n:127.0.0.1:54784_solr 
c:people s:shard2 r:core_node4 x:people_shard2_replica1] 
o.a.s.s.SolrIndexSearcher Opening [Searcher@185ce4c0[people_shard2_replica1] 
main]
   [junit4]   2> 353776 INFO  (qtp1231523479-1176) [n:127.0.0.1:54784_solr 
c:people s:shard2 r:core_node4 x:people_shard2_replica1] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/solrCloudCollectionConfig
   [junit4]   2> 353776 INFO  (qtp1231523479-1176) [n:127.0.0.1:54784_solr 
c:people s:shard2 r:core_node4 x:people_shard2_replica1] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/solrCloudCollectionConfig
   [junit4]   2> 353776 INFO  (qtp1231523479-1176) [n:127.0.0.1:54784_solr 
c:people s:shard2 r:core_node4 x:people_shard2_replica1] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 353848 INFO  (qtp557959012-1151) [n:127.0.0.1:39449_solr 
c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.u.UpdateHandler 
Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 353848 INFO  (qtp557959012-1151) [n:127.0.0.1:39449_solr 
c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.u.UpdateLog 
Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 
maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 353849 INFO  (qtp557959012-1151) [n:127.0.0.1:39449_solr 
c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.u.CommitTracker 
Hard AutoCommit: disabled
   [junit4]   2> 353849 INFO  (qtp557959012-1151) [n:127.0.0.1:39449_solr 
c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.u.CommitTracker 
Soft AutoCommit: disabled
   [junit4]   2> 353849 INFO  (qtp1231523479-1176) [n:127.0.0.1:54784_solr 
c:people s:shard2 r:core_node4 x:people_shard2_replica1] o.a.s.u.UpdateLog 
Could not find max version in index or recent updates, using new clock 
1547296854174072832
   [junit4]   2> 353849 INFO  
(searcherExecutor-460-thread-1-processing-n:127.0.0.1:54784_solr 
x:people_shard2_replica1 s:shard2 c:people r:core_node4) 
[n:127.0.0.1:54784_solr c:people s:shard2 r:core_node4 
x:people_shard2_replica1] o.a.s.c.SolrCore [people_shard2_replica1] Registered 
new searcher Searcher@185ce4c0[people_shard2_replica1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 353859 INFO  (qtp1231523479-1176) [n:127.0.0.1:54784_solr 
c:people s:shard2 r:core_node4 x:people_shard2_replica1] 
o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for 
shard shard2: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 353859 INFO  (qtp557959012-1151) [n:127.0.0.1:39449_solr 
c:people s:shard2 r:core_node1 x:people_shard2_replica2] 
o.a.s.s.SolrIndexSearcher Opening [Searcher@324d3dc3[people_shard2_replica2] 
main]
   [junit4]   2> 353860 INFO  (qtp557959012-1151) [n:127.0.0.1:39449_solr 
c:people s:shard2 r:core_node1 x:people_shard2_replica2] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/solrCloudCollectionConfig
   [junit4]   2> 353861 INFO  (qtp557959012-1151) [n:127.0.0.1:39449_solr 
c:people s:shard2 r:core_node1 x:people_shard2_replica2] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/solrCloudCollectionConfig
   [junit4]   2> 353861 INFO  (qtp557959012-1151) [n:127.0.0.1:39449_solr 
c:people s:shard2 r:core_node1 x:people_shard2_replica2] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 353861 INFO  (qtp557959012-1151) [n:127.0.0.1:39449_solr 
c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.u.UpdateLog 
Could not find max version in index or recent updates, using new clock 
1547296854186655744
   [junit4]   2> 353870 INFO  
(searcherExecutor-459-thread-1-processing-n:127.0.0.1:39449_solr 
x:people_shard2_replica2 s:shard2 c:people r:core_node1) 
[n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 
x:people_shard2_replica2] o.a.s.c.SolrCore [people_shard2_replica2] Registered 
new searcher Searcher@324d3dc3[people_shard2_replica2] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 353960 INFO  
(zkCallback-177-thread-2-processing-n:127.0.0.1:39449_solr) 
[n:127.0.0.1:39449_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/people/state.json] for collection [people] has occurred - 
updating... (live nodes size: [5])
   [junit4]   2> 353960 INFO  
(zkCallback-180-thread-1-processing-n:127.0.0.1:58449_solr) 
[n:127.0.0.1:58449_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/people/state.json] for collection [people] has occurred - 
updating... (live nodes size: [5])
   [junit4]   2> 353960 INFO  
(zkCallback-171-thread-1-processing-n:127.0.0.1:38475_solr) 
[n:127.0.0.1:38475_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/people/state.json] for collection [people] has occurred - 
updating... (live nodes size: [5])
   [junit4]   2> 353961 INFO  
(zkCallback-161-thread-2-processing-n:127.0.0.1:54784_solr) 
[n:127.0.0.1:54784_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/people/state.json] for collection [people] has occurred - 
updating... (live nodes size: [5])
   [junit4]   2> 354263 INFO  (qtp1663009772-1163) [n:127.0.0.1:38475_solr 
c:people s:shard1 r:core_node2 x:people_shard1_replica2] 
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 354263 INFO  (qtp1663009772-1163) [n:127.0.0.1:38475_solr 
c:people s:shard1 r:core_node2 x:people_shard1_replica2] 
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 354263 INFO  (qtp1663009772-1163) [n:127.0.0.1:38475_solr 
c:people s:shard1 r:core_node2 x:people_shard1_replica2] o.a.s.c.SyncStrategy 
Sync replicas to http://127.0.0.1:38475/solr/people_shard1_replica2/
   [junit4]   2> 354263 INFO  (qtp1663009772-1163) [n:127.0.0.1:38475_solr 
c:people s:shard1 r:core_node2 x:people_shard1_replica2] o.a.s.u.PeerSync 
PeerSync: core=people_shard1_replica2 url=http://127.0.0.1:38475/solr START 
replicas=[http://127.0.0.1:58449/solr/people_shard1_replica1/] nUpdates=100
   [junit4]   2> 354265 INFO  (qtp1663009772-1163) [n:127.0.0.1:38475_solr 
c:people s:shard1 r:core_node2 x:people_shard1_replica2] o.a.s.u.PeerSync 
PeerSync: core=people_shard1_replica2 url=http://127.0.0.1:38475/solr DONE.  We 
have no versions.  sync failed.
   [junit4]   2> 354311 INFO  (qtp1201973810-1187) [n:127.0.0.1:58449_solr 
c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.c.S.Request 
[people_shard1_replica1]  webapp=/solr path=/get 
params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2}
 status=0 QTime=1
   [junit4]   2> 354312 INFO  (qtp1663009772-1163) [n:127.0.0.1:38475_solr 
c:people s:shard1 r:core_node2 x:people_shard1_replica2] o.a.s.c.SyncStrategy 
Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 354312 INFO  (qtp1663009772-1163) [n:127.0.0.1:38475_solr 
c:people s:shard1 r:core_node2 x:people_shard1_replica2] 
o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we 
can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 354339 INFO  (qtp1663009772-1163) [n:127.0.0.1:38475_solr 
c:people s:shard1 r:core_node2 x:people_shard1_replica2] 
o.a.s.c.ShardLeaderElectionContext I am the new leader: 
http://127.0.0.1:38475/solr/people_shard1_replica2/ shard1
   [junit4]   2> 354362 INFO  (qtp1231523479-1176) [n:127.0.0.1:54784_solr 
c:people s:shard2 r:core_node4 x:people_shard2_replica1] 
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 354362 INFO  (qtp1231523479-1176) [n:127.0.0.1:54784_solr 
c:people s:shard2 r:core_node4 x:people_shard2_replica1] 
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 354362 INFO  (qtp1231523479-1176) [n:127.0.0.1:54784_solr 
c:people s:shard2 r:core_node4 x:people_shard2_replica1] o.a.s.c.SyncStrategy 
Sync replicas to http://127.0.0.1:54784/solr/people_shard2_replica1/
   [junit4]   2> 354362 INFO  (qtp1231523479-1176) [n:127.0.0.1:54784_solr 
c:people s:shard2 r:core_node4 x:people_shard2_replica1] o.a.s.u.PeerSync 
PeerSync: core=people_shard2_replica1 url=http://127.0.0.1:54784/solr START 
replicas=[http://127.0.0.1:39449/solr/people_shard2_replica2/] nUpdates=100
   [junit4]   2> 354362 INFO  (qtp1231523479-1176) [n:127.0.0.1:54784_solr 
c:people s:shard2 r:core_node4 x:people_shard2_replica1] o.a.s.u.PeerSync 
PeerSync: core=people_shard2_replica1 url=http://127.0.0.1:54784/solr DONE.  We 
have no versions.  sync failed.
   [junit4]   2> 354374 INFO  (qtp557959012-1150) [n:127.0.0.1:39449_solr 
c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.c.S.Request 
[people_shard2_replica2]  webapp=/solr path=/get 
params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2}
 status=0 QTime=8
   [junit4]   2> 354374 INFO  (qtp1231523479-1176) [n:127.0.0.1:54784_solr 
c:people s:shard2 r:core_node4 x:people_shard2_replica1] o.a.s.c.SyncStrategy 
Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 354374 INFO  (qtp1231523479-1176) [n:127.0.0.1:54784_solr 
c:people s:shard2 r:core_node4 x:people_shard2_replica1] 
o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we 
can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 354387 INFO  (qtp1231523479-1176) [n:127.0.0.1:54784_solr 
c:people s:shard2 r:core_node4 x:people_shard2_replica1] 
o.a.s.c.ShardLeaderElectionContext I am the new leader: 
http://127.0.0.1:54784/solr/people_shard2_replica1/ shard2
   [junit4]   2> 354493 INFO  
(zkCallback-177-thread-2-processing-n:127.0.0.1:39449_solr) 
[n:127.0.0.1:39449_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/people/state.json] for collection [people] has occurred - 
updating... (live nodes size: [5])
   [junit4]   2> 354493 INFO  
(zkCallback-171-thread-1-processing-n:127.0.0.1:38475_solr) 
[n:127.0.0.1:38475_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/people/state.json] for collection [people] has occurred - 
updating... (live nodes size: [5])
   [junit4]   2> 354493 INFO  
(zkCallback-180-thread-1-processing-n:127.0.0.1:58449_solr) 
[n:127.0.0.1:58449_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/people/state.json] for collection [people] has occurred - 
updating... (live nodes size: [5])
   [junit4]   2> 354494 INFO  
(zkCallback-161-thread-1-processing-n:127.0.0.1:54784_solr) 
[n:127.0.0.1:54784_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/people/state.json] for collection [people] has occurred - 
updating... (live nodes size: [5])
   [junit4]   2> 354495 INFO  (qtp1663009772-1163) [n:127.0.0.1:38475_solr 
c:people s:shard1 r:core_node2 x:people_shard1_replica2] o.a.s.c.ZkController I 
am the leader, no recovery necessary
   [junit4]   2> 354502 INFO  (qtp1663009772-1163) [n:127.0.0.1:38475_solr 
c:people s:shard1 r:core_node2 x:people_shard1_replica2] o.a.s.s.HttpSolrCall 
[admin] webapp=null path=/admin/cores 
params={property.schema=schema-docValuesJoin.xml&property.config=solrconfig-doctransformers.xml&qt=/admin/cores&collection.configName=solrCloudCollectionConfig&name=people_shard1_replica2&action=CREATE&numShards=2&collection=people&shard=shard1&wt=javabin&version=2}
 status=0 QTime=2210
   [junit4]   2> 354545 INFO  (qtp1231523479-1176) [n:127.0.0.1:54784_solr 
c:people s:shard2 r:core_node4 x:people_shard2_replica1] o.a.s.c.ZkController I 
am the leader, no recovery necessary
   [junit4]   2> 354551 INFO  (qtp1231523479-1176) [n:127.0.0.1:54784_solr 
c:people s:shard2 r:core_node4 x:people_shard2_replica1] o.a.s.s.HttpSolrCall 
[admin] webapp=null path=/admin/cores 
params={property.schema=schema-docValuesJoin.xml&property.config=solrconfig-doctransformers.xml&qt=/admin/cores&collection.configName=solrCloudCollectionConfig&name=people_shard2_replica1&action=CREATE&numShards=2&collection=people&shard=shard2&wt=javabin&version=2}
 status=0 QTime=2259
   [junit4]   2> 354695 INFO  
(zkCallback-177-thread-2-processing-n:127.0.0.1:39449_solr) 
[n:127.0.0.1:39449_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/people/state.json] for collection [people] has occurred - 
updating... (live nodes size: [5])
   [junit4]   2> 354695 INFO  
(zkCallback-171-thread-1-processing-n:127.0.0.1:38475_solr) 
[n:127.0.0.1:38475_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/people/state.json] for collection [people] has occurred - 
updating... (live nodes size: [5])
   [junit4]   2> 354696 INFO  
(zkCallback-180-thread-1-processing-n:127.0.0.1:58449_solr) 
[n:127.0.0.1:58449_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/people/state.json] for collection [people] has occurred - 
updating... (live nodes size: [5])
   [junit4]   2> 354697 INFO  
(zkCallback-161-thread-1-processing-n:127.0.0.1:54784_solr) 
[n:127.0.0.1:54784_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/people/state.json] for collection [people] has occurred - 
updating... (live nodes size: [5])
   [junit4]   2> 354778 INFO  (qtp1201973810-1191) [n:127.0.0.1:58449_solr 
c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.c.ZkController 
Core needs to recover:people_shard1_replica1
   [junit4]   2> 354779 INFO  (qtp1201973810-1191) [n:127.0.0.1:58449_solr 
c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.s.HttpSolrCall 
[admin] webapp=null path=/admin/cores 
params={property.schema=schema-docValuesJoin.xml&property.config=solrconfig-doctransformers.xml&qt=/admin/cores&collection.configName=solrCloudCollectionConfig&name=people_shard1_replica1&action=CREATE&numShards=2&collection=people&shard=shard1&wt=javabin&version=2}
 status=0 QTime=2489
   [junit4]   2> 354780 INFO  
(updateExecutor-163-thread-1-processing-n:127.0.0.1:58449_solr 
x:people_shard1_replica1 s:shard1 c:people r:core_node3) 
[n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 
x:people_shard1_replica1] o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 354785 INFO  
(recoveryExecutor-164-thread-1-processing-n:127.0.0.1:58449_solr 
x:people_shard1_replica1 s:shard1 c:people r:core_node3) 
[n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 
x:people_shard1_replica1] o.a.s.c.RecoveryStrategy Starting recovery process. 
recoveringAfterStartup=true
   [junit4]   2> 354786 INFO  
(recoveryExecutor-164-thread-1-processing-n:127.0.0.1:58449_solr 
x:people_shard1_replica1 s:shard1 c:people r:core_node3) 
[n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 
x:people_shard1_replica1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
   [junit4]   2> 354786 INFO  
(recoveryExecutor-164-thread-1-processing-n:127.0.0.1:58449_solr 
x:people_shard1_replica1 s:shard1 c:people r:core_node3) 
[n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 
x:people_shard1_replica1] o.a.s.c.RecoveryStrategy Begin buffering updates. 
core=[people_shard1_replica1]
   [junit4]   2> 354786 INFO  
(recoveryExecutor-164-thread-1-processing-n:127.0.0.1:58449_solr 
x:people_shard1_replica1 s:shard1 c:people r:core_node3) 
[n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 
x:people_shard1_replica1] o.a.s.u.UpdateLog Starting to buffer updates. 
FSUpdateLog{state=ACTIVE, tlog=null}
   [junit4]   2> 354786 INFO  
(recoveryExecutor-164-thread-1-processing-n:127.0.0.1:58449_solr 
x:people_shard1_replica1 s:shard1 c:people r:core_node3) 
[n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 
x:people_shard1_replica1] o.a.s.c.RecoveryStrategy Publishing state of core 
[people_shard1_replica1] as recovering, leader is 
[http://127.0.0.1:38475/solr/people_shard1_replica2/] and I am 
[http://127.0.0.1:58449/solr/people_shard1_replica1/]
   [junit4]   2> 354797 INFO  
(recoveryExecutor-164-thread-1-processing-n:127.0.0.1:58449_solr 
x:people_shard1_replica1 s:shard1 c:people r:core_node3) 
[n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 
x:people_shard1_replica1] o.a.s.c.RecoveryStrategy Sending prep recovery 
command to [http://127.0.0.1:38475/solr]; [WaitForState: 
action=PREPRECOVERY&core=people_shard1_replica2&nodeName=127.0.0.1:58449_solr&coreNodeName=core_node3&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
   [junit4]   2> 354806 INFO  (qtp1663009772-1161) [n:127.0.0.1:38475_solr    ] 
o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node3, state: 
recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
   [junit4]   2> 354822 INFO  (qtp1663009772-1161) [n:127.0.0.1:38475_solr    ] 
o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see 
people_shard1_replica2 (shard1 of people) have state: recovering
   [junit4]   2> 354822 INFO  (qtp1663009772-1161) [n:127.0.0.1:38475_solr    ] 
o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=people, 
shard=shard1, thisCore=people_shard1_replica2, leaderDoesNotNeedRecovery=false, 
isLeader? true, live=true, checkLive=true, currentState=down, 
localState=active, nodeName=127.0.0.1:58449_solr, coreNodeName=core_node3, 
onlyIfActiveCheckResult=false, nodeProps: 
core_node3:{"core":"people_shard1_replica1","base_url":"http://127.0.0.1:58449/solr","node_name":"127.0.0.1:58449_solr","state":"down"}
   [junit4]   2> 354870 INFO  (qtp557959012-1151) [n:127.0.0.1:39449_solr 
c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.c.ZkController 
Core needs to recover:people_shard2_replica2
   [junit4]   2> 354871 INFO  (qtp557959012-1151) [n:127.0.0.1:39449_solr 
c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.s.HttpSolrCall 
[admin] webapp=null path=/admin/cores 
params={property.schema=schema-docValuesJoin.xml&property.config=solrconfig-doctransformers.xml&qt=/admin/cores&collection.configName=solrCloudCollectionConfig&name=people_shard2_replica2&action=CREATE&numShards=2&collection=people&shard=shard2&wt=javabin&version=2}
 status=0 QTime=2608
   [junit4]   2> 354905 INFO  
(updateExecutor-165-thread-1-processing-n:127.0.0.1:39449_solr 
x:people_shard2_replica2 s:shard2 c:people r:core_node1) 
[n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 
x:people_shard2_replica2] o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 354906 INFO  
(zkCallback-171-thread-1-processing-n:127.0.0.1:38475_solr) 
[n:127.0.0.1:38475_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/people/state.json] for collection [people] has occurred - 
updating... (live nodes size: [5])
   [junit4]   2> 354906 INFO  
(zkCallback-180-thread-1-processing-n:127.0.0.1:58449_solr) 
[n:127.0.0.1:58449_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/people/state.json] for collection [people] has occurred - 
updating... (live nodes size: [5])
   [junit4]   2> 354907 INFO  
(zkCallback-177-thread-2-processing-n:127.0.0.1:39449_solr) 
[n:127.0.0.1:39449_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/people/state.json] for collection [people] has occurred - 
updating... (live nodes size: [5])
   [junit4]   2> 354908 INFO  
(zkCallback-161-thread-2-processing-n:127.0.0.1:54784_solr) 
[n:127.0.0.1:54784_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/people/state.json] for collection [people] has occurred - 
updating... (live nodes size: [5])
   [junit4]   2> 354912 INFO  (qtp557959012-1153) [n:127.0.0.1:39449_solr    ] 
o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 
30 seconds. Check all shard replicas
   [junit4]   2> 354917 INFO  
(recoveryExecutor-166-thread-1-processing-n:127.0.0.1:39449_solr 
x:people_shard2_replica2 s:shard2 c:people r:core_node1) 
[n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 
x:people_shard2_replica2] o.a.s.c.RecoveryStrategy Starting recovery process. 
recoveringAfterStartup=true
   [junit4]   2> 354918 INFO  
(recoveryExecutor-166-thread-1-processing-n:127.0.0.1:39449_solr 
x:people_shard2_replica2 s:shard2 c:people r:core_node1) 
[n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 
x:people_shard2_replica2] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
   [junit4]   2> 354918 INFO  
(recoveryExecutor-166-thread-1-processing-n:127.0.0.1:39449_solr 
x:people_shard2_replica2 s:shard2 c:people r:core_node1) 
[n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 
x:people_shard2_replica2] o.a.s.c.RecoveryStrategy Begin buffering updates. 
core=[people_shard2_replica2]
   [junit4]   2> 354918 INFO  
(recoveryExecutor-166-thread-1-processing-n:127.0.0.1:39449_solr 
x:people_shard2_replica2 s:shard2 c:people r:core_node1) 
[n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 
x:people_shard2_replica2] o.a.s.u.UpdateLog Starting to buffer updates. 
FSUpdateLog{state=ACTIVE, tlog=null}
   [junit4]   2> 354918 INFO  
(recoveryExecutor-166-thread-1-processing-n:127.0.0.1:39449_solr 
x:people_shard2_replica2 s:shard2 c:people r:core_node1) 
[n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 
x:people_shard2_replica2] o.a.s.c.RecoveryStrategy Publishing state of core 
[people_shard2_replica2] as recovering, leader is 
[http://127.0.0.1:54784/solr/people_shard2_replica1/] and I am 
[http://127.0.0.1:39449/solr/people_shard2_replica2/]
   [junit4]   2> 354919 INFO  
(OverseerCollectionConfigSetProcessor-96706052996988935-127.0.0.1:54784_solr-n_0000000000)
 [n:127.0.0.1:54784_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: 
/overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may 
have disconnected from ZooKeeper
   [junit4]   2> 354931 INFO  
(recoveryExecutor-166-thread-1-processing-n:127.0.0.1:39449_solr 
x:people_shard2_replica2 s:shard2 c:people r:core_node1) 
[n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 
x:people_shard2_replica2] o.a.s.c.RecoveryStrategy Sending prep recovery 
command to [http://127.0.0.1:54784/solr]; [WaitForState: 
action=PREPRECOVERY&core=people_shard2_replica1&nodeName=127.0.0.1:39449_solr&coreNodeName=core_node1&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
   [junit4]   2> 354959 INFO  (qtp1231523479-1177) [n:127.0.0.1:54784_solr    ] 
o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node1, state: 
recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
   [junit4]   2> 354960 INFO  (qtp1231523479-1177) [n:127.0.0.1:54784_solr    ] 
o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see 
people_shard2_replica1 (shard2 of people) have state: recovering
   [junit4]   2> 354960 INFO  (qtp1231523479-1177) [n:127.0.0.1:54784_solr    ] 
o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=people, 
shard=shard2, thisCore=people_shard2_replica1, leaderDoesNotNeedRecovery=false, 
isLeader? true, live=true, checkLive=true, currentState=down, 
localState=active, nodeName=127.0.0.1:39449_solr, coreNodeName=core_node1, 
onlyIfActiveCheckResult=false, nodeProps: 
core_node1:{"core":"people_shard2_replica2","base_url":"http://127.0.0.1:39449/solr","node_name":"127.0.0.1:39449_solr","state":"down"}
   [junit4]   2> 355023 INFO  
(zkCallback-177-thread-2-processing-n:127.0.0.1:39449_solr) 
[n:127.0.0.1:39449_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/people/state.json] for collection [people] has occurred - 
updating... (live nodes size: [5])
   [junit4]   2> 355023 INFO  
(zkCallback-171-thread-1-processing-n:127.0.0.1:38475_solr) 
[n:127.0.0.1:38475_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/people/state.json] for collection [people] has occurred - 
updating... (live nodes size: [5])
   [junit4]   2> 355024 INFO  
(zkCallback-180-thread-1-processing-n:127.0.0.1:58449_solr) 
[n:127.0.0.1:58449_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/people/state.json] for collection [people] has occurred - 
updating... (live nodes size: [5])
   [junit4]   2> 355025 INFO  
(zkCallback-161-thread-2-processing-n:127.0.0.1:54784_solr) 
[n:127.0.0.1:54784_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/people/state.json] for collection [people] has occurred - 
updating... (live nodes size: [5])
   [junit4]   2> 355822 INFO  (qtp1663009772-1161) [n:127.0.0.1:38475_solr    ] 
o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=people, 
shard=shard1, thisCore=people_shard1_replica2, leaderDoesNotNeedRecovery=false, 
isLeader? true, live=true, checkLive=true, currentState=recovering, 
localState=active, nodeName=127.0.0.1:58449_solr, coreNodeName=core_node3, 
onlyIfActiveCheckResult=false, nodeProps: 
core_node3:{"core":"people_shard1_replica1","base_url":"http://127.0.0.1:58449/solr","node_name":"127.0.0.1:58449_solr","state":"recovering"}
   [junit4]   2> 355822 INFO  (qtp1663009772-1161) [n:127.0.0.1:38475_solr    ] 
o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node3, state: recovering, 
checkLive: true, onlyIfLeader: true for: 1 seconds.
   [junit4]   2> 355822 INFO  (qtp1663009772-1161) [n:127.0.0.1:38475_solr    ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores 
params={nodeName=127.0.0.1:58449_solr&onlyIfLeaderActive=true&core=people_shard1_replica2&coreNodeName=core_node3&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2}
 status=0 QTime=1016
   [junit4]   2> 355966 INFO  (qtp1231523479-1177) [n:127.0.0.1:54784_solr    ] 
o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=people, 
shard=shard2, thisCore=people_shard2_replica1, leaderDoesNotNeedRecovery=false, 
isLeader? true, live=true, checkLive=true, currentState=recovering, 
localState=active, nodeName=127.0.0.1:39449_solr, coreNodeName=core_node1, 
onlyIfActiveCheckResult=false, nodeProps: 
core_node1:{"core":"people_shard2_replica2","base_url":"http://127.0.0.1:39449/solr","node_name":"127.0.0.1:39449_solr","state":"recovering"}
   [junit4]   2> 355966 INFO  (qtp1231523479-1177) [n:127.0.0.1:54784_solr    ] 
o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node1, state: recovering, 
checkLive: true, onlyIfLeader: true for: 1 seconds.
   [junit4]   2> 355966 INFO  (qtp1231523479-1177) [n:127.0.0.1:54784_solr    ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores 
params={nodeName=127.0.0.1:39449_solr&onlyIfLeaderActive=true&core=people_shard2_replica1&coreNodeName=core_node1&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2}
 status=0 QTime=1006
   [junit4]   2> 362834 INFO  
(recoveryExecutor-164-thread-1-processing-n:127.0.0.1:58449_solr 
x:people_shard1_replica1 s:shard1 c:people r:core_node3) 
[n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 
x:people_shard1_replica1] o.a.s.c.RecoveryStrategy Attempting to PeerSync from 
[http://127.0.0.1:38475/solr/people_shard1_replica2/] - 
recoveringAfterStartup=[true]
   [junit4]   2> 362834 INFO  
(recoveryExecutor-164-thread-1-processing-n:127.0.0.1:58449_solr 
x:people_shard1_replica1 s:shard1 c:people r:core_node3) 
[n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 
x:people_shard1_replica1] o.a.s.u.PeerSync PeerSync: 
core=people_shard1_replica1 url=http://127.0.0.1:58449/solr START 
replicas=[http://127.0.0.1:38475/solr/people_shard1_replica2/] nUpdates=100
   [junit4]   2> 362857 INFO  (qtp1663009772-1165) [n:127.0.0.1:38475_solr 
c:people s:shard1 r:core_node2 x:people_shard1_replica2] 
o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 
result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, 
maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
   [junit4]   2> 362858 INFO  (qtp1663009772-1165) [n:127.0.0.1:38475_solr 
c:people s:shard1 r:core_node2 x:people_shard1_replica2] o.a.s.c.S.Request 
[people_shard1_replica2]  webapp=/solr path=/get 
params={distrib=false&qt=/get&getFingerprint=9223372036854775807&wt=javabin&version=2}
 status=0 QTime=2
   [junit4]   2> 362863 INFO  
(recoveryExecutor-164-thread-1-processing-n:127.0.0.1:58449_solr 
x:people_shard1_replica1 s:shard1 c:people r:core_node3) 
[n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 
x:people_shard1_replica1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 
result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, 
maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
   [junit4]   2> 362863 INFO  
(recoveryExecutor-164-thread-1-processing-n:127.0.0.1:58449_solr 
x:people_shard1_replica1 s:shard1 c:people r:core_node3) 
[n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 
x:people_shard1_replica1] o.a.s.u.PeerSync We are already in sync. No need to 
do a PeerSync 
   [junit4]   2> 362863 INFO  
(recoveryExecutor-164-thread-1-processing-n:127.0.0.1:58449_solr 
x:people_shard1_replica1 s:shard1 c:people r:core_node3) 
[n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 
x:people_shard1_replica1] o.a.s.u.DirectUpdateHandler2 start 
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 362863 INFO  
(recoveryExecutor-164-thread-1-processing-n:127.0.0.1:58449_solr 
x:people_shard1_replica1 s:shard1 c:people r:core_node3) 
[n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 
x:people_shard1_replica1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. 
Skipping IW.commit.
   [junit4]   2> 362874 INFO  
(recoveryExecutor-164-thread-1-processing-n:127.0.0.1:58449_solr 
x:people_shard1_replica1 s:shard1 c:people r:core_node3) 
[n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 
x:people_shard1_replica1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 362881 INFO  
(recoveryExecutor-164-thread-1-processing-n:127.0.0.1:58449_solr 
x:people_shard1_replica1 s:shard1 c:people r:core_node3) 
[n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 
x:people_shard1_replica1] o.a.s.c.RecoveryStrategy PeerSync stage of recovery 
was successful.
   [junit4]   2> 362882 INFO  
(recoveryExecutor-164-thread-1-processing-n:127.0.0.1:58449_solr 
x:people_shard1_replica1 s:shard1 c:people r:core_node3) 
[n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 
x:people_shard1_replica1] o.a.s.c.RecoveryStrategy Replaying updates buffered 
during PeerSync.
   [junit4]   2> 362882 INFO  
(recoveryExecutor-164-thread-1-processing-n:127.0.0.1:58449_solr 
x:people_shard1_replica1 s:shard1 c:people r:core_node3) 
[n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 
x:people_shard1_replica1] o.a.s.c.RecoveryStrategy No replay needed.
   [junit4]   2> 362882 INFO  
(recoveryExecutor-164-thread-1-processing-n:127.0.0.1:58449_solr 
x:people_shard1_replica1 s:shard1 c:people r:core_node3) 
[n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 
x:people_shard1_replica1] o.a.s.c.RecoveryStrategy Registering as Active after 
recovery.
   [junit4]   2> 362886 INFO  
(zkCallback-177-thread-2-processing-n:127.0.0.1:39449_solr) 
[n:127.0.0.1:39449_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/people/state.json] for collection [people] has occurred - 
updating... (live nodes size: [5])
   [junit4]   2> 362886 INFO  
(zkCallback-171-thread-1-processing-n:127.0.0.1:38475_solr) 
[n:127.0.0.1:38475_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/people/state.json] for collection [people] has occurred - 
updating... (live nodes size: [5])
   [junit4]   2> 362886 INFO  
(zkCallback-180-thread-1-processing-n:127.0.0.1:58449_solr) 
[n:127.0.0.1:58449_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/people/state.json] for collection [people] has occurred - 
updating... (live nodes size: [5])
   [junit4]   2> 362888 INFO  
(zkCallback-161-thread-1-processing-n:127.0.0.1:54784_solr) 
[n:127.0.0.1:54784_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/people/state.json] for collection [people] has occurred - 
updating... (live nodes size: [5])
   [junit4]   2> 362973 INFO  
(recoveryExecutor-166-thread-1-processing-n:127.0.0.1:39449_solr 
x:people_shard2_replica2 s:shard2 c:people r:core_node1) 
[n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 
x:people_shard2_replica2] o.a.s.c.RecoveryStrategy Attempting to PeerSync from 
[http://127.0.0.1:54784/solr/people_shard2_replica1/] - 
recoveringAfterStartup=[true]
   [junit4]   2> 362973 INFO  
(recoveryExecutor-166-thread-1-processing-n:127.0.0.1:39449_solr 
x:people_shard2_replica2 s:shard2 c:people r:core_node1) 
[n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 
x:people_shard2_replica2] o.a.s.u.PeerSync PeerSync: 
core=people_shard2_replica2 url=http://127.0.0.1:39449/solr START 
replicas=[http://127.0.0.1:54784/solr/people_shard2_replica1/] nUpdates=100
   [junit4]   2> 362983 INFO  (qtp1231523479-1172) [n:127.0.0.1:54784_solr 
c:people s:shard2 r:core_node4 x:people_shard2_replica1] 
o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 
result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, 
maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
   [junit4]   2> 362983 INFO  (qtp1231523479-1172) [n:127.0.0.1:54784_solr 
c:people s:shard2 r:core_node4 x:people_shard2_replica1] o.a.s.c.S.Request 
[people_shard2_replica1]  webapp=/solr path=/get 
params={distrib=false&qt=/get&getFingerprint=9223372036854775807&wt=javabin&version=2}
 status=0 QTime=3
   [junit4]   2> 362991 INFO  
(recoveryExecutor-166-thread-1-processing-n:127.0.0.1:39449_solr 
x:people_shard2_replica2 s:shard2 c:people r:core_node1) 
[n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 
x:people_shard2_replica2] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 
result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, 
maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
   [junit4]   2> 362991 INFO  
(recoveryExecutor-166-thread-1-processing-n:127.0.0.1:39449_solr 
x:people_shard2_replica2 s:shard2 c:people r:core_node1) 
[n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 
x:people_shard2_replica2] o.a.s.u.PeerSync We are already in sync. No need to 
do a PeerSync 
   [junit4]   2> 362991 INFO  
(recoveryExecutor-166-thread-1-processing-n:127.0.0.1:39449_solr 
x:people_shard2_replica2 s:shard2 c:people r:core_node1) 
[n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 
x:people_shard2_replica2] o.a.s.u.DirectUpdateHandler2 start 
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 362991 INFO  
(recoveryExecutor-166-thread-1-processing-n:127.0.0.1:39449_solr 
x:people_shard2_replica2 s:shard2 c:people r:core_node1) 
[n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 
x:people_shard2_replica2] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. 
Skipping IW.commit.
   [junit4]   2> 362997 INFO  
(recoveryExecutor-166-thread-1-processing-n:127.0.0.1:39449_solr 
x:people_shard2_replica2 s:shard2 c:people r:core_node1) 
[n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 
x:people_shard2_replica2] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 363001 INFO  
(recoveryExecutor-166-thread-1-processing-n:127.0.0.1:39449_solr 
x:people_shard2_replica2 s:shard2 c:people r:core_node1) 
[n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 
x:people_shard2_replica2] o.a.s.c.RecoveryStrategy PeerSync stage of recovery 
was successful.
   [junit4]   2> 363001 INFO  
(recoveryExecutor-166-thread-1-processing-n:127.0.0.1:39449_solr 
x:people_shard2_replica2 s:shard2 c:people r:core_node1) 
[n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 
x:people_shard2_replica2] o.a.s.c.RecoveryStrategy Replaying updates buffered 
during PeerSync.
   [junit4]   2> 363001 INFO  
(recoveryExecutor-166-thread-1-processing-n:127.0.0.1:39449_solr 
x:people_shard2_replica2 s:shard2 c:people r:core_node1) 
[n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 
x:people_shard2_replica2] o.a.s.c.RecoveryStrategy No replay needed.
   [junit4]   2> 363002 INFO  
(recoveryExecutor-166-thread-1-processing-n:127.0.0.1:39449_solr 
x:people_shard2_replica2 s:shard2 c:people r:core_node1) 
[n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 
x:people_shard2_replica2] o.a.s.c.RecoveryStrategy Registering as Active after 
recovery.
   [junit4]   2> 363108 INFO  
(zkCallback-177-thread-2-processing-n:127.0.0.1:39449_solr) 
[n:127.0.0.1:39449_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/people/state.json] for collection [people] has occurred - 
updating... (live nodes size: [5])
   [junit4]   2> 363108 INFO  
(zkCallback-171-thread-1-processing-n:127.0.0.1:38475_solr) 
[n:127.0.0.1:38475_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/people/state.json] for collection [people] has occurred - 
updating... (live nodes size: [5])
   [junit4]   2> 363108 INFO  
(zkCallback-180-thread-1-processing-n:127.0.0.1:58449_solr) 
[n:127.0.0.1:58449_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/people/state.json] for collection [people] has occurred - 
updating... (live nodes size: [5])
   [junit4]   2> 363108 INFO  
(zkCallback-161-thread-1-processing-n:127.0.0.1:54784_solr) 
[n:127.0.0.1:54784_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/people/state.json] for collection [people] has occurred - 
updating... (live nodes size: [5])
   [junit4]   2> 363937 INFO  (qtp557959012-1153) [n:127.0.0.1:39449_solr    ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={replicationFactor=2&property.schema=schema-docValuesJoin.xml&property.config=solrconfig-doctransformers.xml&collection.configName=solrCloudCollectionConfig&name=people&action=CREATE&numShards=2&wt=javabin&version=2}
 status=0 QTime=11864
   [junit4]   2> 363942 INFO  (qtp557959012-1154) [n:127.0.0.1:39449_solr    ] 
o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params 
replicationFactor=2&property.schema=schema-docValuesJoin.xml&property.config=solrconfig-doctransformers.xml&collection.configName=solrCloudCollectionConfig&name=departments&action=CREATE&numShards=2&wt=javabin&version=2
 and sendToOCPQueue=true
   [junit4]   2> 363946 INFO  
(OverseerThreadFactory-448-thread-2-processing-n:127.0.0.1:54784_solr) 
[n:127.0.0.1:54784_solr    ] o.a.s.c.CreateCollectionCmd Create collection 
departments
   [junit4]   2> 364382 INFO  (qtp1663009772-1164) [n:127.0.0.1:38475_solr    ] 
o.a.s.h.a.CoreAdminOperation core create command 
property.schema=schema-docValuesJoin.xml&property.config=solrconfig-doctransformers.xml&qt=/admin/cores&collection.configName=solrCloudCollectionConfig&name=departments_shard2_replica1&action=CREATE&numShards=2&collection=departments&shard=shard2&wt=javabin&version=2
   [junit4]   2> 364383 INFO  (qtp557959012-1156) [n:127.0.0.1:39449_solr    ] 
o.a.s.h.a.CoreAdminOperation core create command 
property.schema=schema-docValuesJoin.xml&property.config=solrconfig-doctransformers.xml&qt=/admin/cores&collection.configName=solrCloudCollectionConfig&name=departments_shard2_replica2&action=CREATE&numShards=2&collection=departments&shard=shard2&wt=javabin&version=2
   [junit4]   2> 364385 INFO  (qtp1231523479-1175) [n:127.0.0.1:54784_solr    ] 
o.a.s.h.a.CoreAdminOperation core create command 
property.schema=schema-docValuesJoin.xml&property.config=solrconfig-doctransformers.xml&qt=/admin/cores&collection.configName=solrCloudCollectionConfig&name=departments_shard1_replica2&action=CREATE&numShards=2&collection=departments&shard=shard1&wt=javabin&version=2
   [junit4]   2> 364399 INFO  (qtp444453979-1197) [n:127.0.0.1:34130_solr    ] 
o.a.s.h.a.CoreAdminOperation core create command 
property.schema=schema-docValuesJoin.xml&property.config=solrconfig-doctransformers.xml&qt=/admin/cores&collection.configName=solrCloudCollectionConfig&name=departments_shard1_replica1&action=CREATE&numShards=2&collection=departments&shard=shard1&wt=javabin&version=2
   [junit4]   2> 364531 INFO  
(zkCallback-161-thread-2-processing-n:127.0.0.1:54784_solr) 
[n:127.0.0.1:54784_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/departments/state.json] for collection [departments] has 
occurred - updating... (live nodes size: [5])
   [junit4]   2> 364532 INFO  
(zkCallback-178-thread-1-processing-n:127.0.0.1:34130_solr) 
[n:127.0.0.1:34130_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/departments/state.json] for collection [departments] has 
occurred - updating... (live nodes size: [5])
   [junit4]   2> 364532 INFO  
(zkCallback-177-thread-2-processing-n:127.0.0.1:39449_solr) 
[n:127.0.0.1:39449_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/departments/state.json] for collection [departments] has 
occurred - updating... (live nodes size: [5])
   [junit4]   2> 364532 INFO  
(zkCallback-171-thread-1-processing-n:127.0.0.1:38475_solr) 
[n:127.0.0.1:38475_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/departments/state.json] for collection [departments] has 
occurred - updating... (live nodes size: [5])
   [junit4]   2> 365449 INFO  (qtp1231523479-1175) [n:127.0.0.1:54784_solr 
c:departments s:shard1  x:departments_shard1_replica2] o.a.s.c.SolrConfig Using 
Lucene MatchVersion: 6.3.0
   [junit4]   2> 365479 INFO  (qtp557959012-1156) [n:127.0.0.1:39449_solr 
c:departments s:shard2  x:departments_shard2_replica2] o.a.s.c.SolrConfig Using 
Lucene MatchVersion: 6.3.0
   [junit4]   2> 365519 INFO  (qtp444453979-1197) [n:127.0.0.1:34130_solr 
c:departments s:shard1  x:departments_shard1_replica1] o.a.s.c.SolrConfig Using 
Lucene MatchVersion: 6.3.0
   [junit4]   2> 365524 INFO  (qtp1663009772-1164) [n:127.0.0.1:38475_solr 
c:departments s:shard2  x:departments_shard2_replica1] o.a.s.c.SolrConfig Using 
Lucene MatchVersion: 6.3.0
   [junit4]   2> 365535 INFO  (qtp1231523479-1175) [n:127.0.0.1:54784_solr 
c:departments s:shard1  x:departments_shard1_replica2] o.a.s.s.IndexSchema 
[departments_shard1_replica2] Schema name=doc-values-for-Join
   [junit4]   2> 365678 WARN  (qtp1231523479-1175) [n:127.0.0.1:54784_solr 
c:departments s:shard1  x:departments_shard1_replica2] o.a.s.s.IndexSchema 
[departments_shard1_replica2] default search field in schema is id. WARNING: 
Deprecated, please use 'df' on request instead.
   [junit4]   2> 365679 INFO  (qtp1231523479-1175) [n:127.0.0.1:54784_solr 
c:departments s:shard1  x:departments_shard1_replica2] o.a.s.s.IndexSchema 
Loaded schema doc-values-for-Join/1.6 with uniqueid field id
   [junit4]   2> 365679 INFO  (qtp557959012-1156) [n:127.0.0.1:39449_solr 
c:departments s:shard2  x:departments_shard2_replica2] o.a.s.s.IndexSchema 
[departments_shard2_replica2] Schema name=doc-values-for-Join
   [junit4]   2> 365687 WARN  (qtp557959012-1156) [n:127.0.0.1:39449_solr 
c:departments s:shard2  x:departments_shard2_replica2] o.a.s.s.IndexSchema 
[departments_shard2_replica2] default search field in schema is id. WARNING: 
Deprecated, please use 'df' on request instead.
   [junit4]   2> 365688 INFO  (qtp557959012-1156) [n:127.0.0.1:39449_solr 
c:departments s:shard2  x:departments_shard2_replica2] o.a.s.s.IndexSchema 
Loaded schema doc-values-for-Join/1.6 with uniqueid field id
   [junit4]   2> 365695 INFO  (qtp1663009772-1164) [n:127.0.0.1:38475_solr 
c:departments s:shard2  x:departments_shard2_replica1] o.a.s.s.IndexSchema 
[departments_shard2_replica1] Schema name=doc-values-for-Join
   [junit4]   2> 365707 INFO  (qtp444453979-1197) [n:127.0.0.1:34130_solr 
c:departments s:shard1  x:departments_shard1_replica1] o.a.s.s.IndexSchema 
[departments_shard1_replica1] Schema name=doc-values-for-Join
   [junit4]   2> 365709 INFO  (qtp1231523479-1175) [n:127.0.0.1:54784_solr 
c:departments s:shard1  x:departments_shard1_replica2] o.a.s.c.CoreContainer 
Creating SolrCore 'departments_shard1_replica2' using configuration from 
collection departments
   [junit4]   2> 365709 INFO  (qtp1231523479-1175) [n:127.0.0.1:54784_solr 
c:departments s:shard1 r:core_node2 x:departments_shard1_replica2] 
o.a.s.c.SolrCore [[departments_shard1_replica2] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node2/departments_shard1_replica2],
 
dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node2/./departments_shard1_replica2/data/]
   [junit4]   2> 365722 INFO  (qtp557959012-1156) [n:127.0.0.1:39449_solr 
c:departments s:shard2  x:departments_shard2_replica2] o.a.s.c.CoreContainer 
Creating SolrCore 'departments_shard2_replica2' using configuration from 
collection departments
   [junit4]   2> 365722 INFO  (qtp557959012-1156) [n:127.0.0.1:39449_solr 
c:departments s:shard2 r:core_node1 x:departments_shard2_replica2] 
o.a.s.c.SolrCore [[departments_shard2_replica2] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node3/departments_shard2_replica2],
 
dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node3/./departments_shard2_replica2/data/]
   [junit4]   2> 365729 WARN  (qtp1663009772-1164) [n:127.0.0.1:38475_solr 
c:departments s:shard2  x:departments_shard2_replica1] o.a.s.s.IndexSchema 
[departments_shard2_replica1] default search field in schema is id. WARNING: 
Deprecated, please use 'df' on request instead.
   [junit4]   2> 365729 INFO  (qtp1663009772-1164) [n:127.0.0.1:38475_solr 
c:departments s:shard2  x:departments_shard2_replica1] o.a.s.s.IndexSchema 
Loaded schema doc-values-for-Join/1.6 with uniqueid field id
   [junit4]   2> 365732 WARN  (qtp444453979-1197) [n:127.0.0.1:34130_solr 
c:departments s:shard1  x:departments_shard1_replica1] o.a.s.s.IndexSchema 
[departments_shard1_replica1] default search field in schema is id. WARNING: 
Deprecated, please use 'df' on request instead.
   [junit4]   2> 365733 INFO  (qtp444453979-1197) [n:127.0.0.1:34130_solr 
c:departments s:shard1  x:departments_shard1_replica1] o.a.s.s.IndexSchema 
Loaded schema doc-values-for-Join/1.6 with uniqueid field id
   [junit4]   2> 365804 INFO  (qtp1663009772-1164) [n:127.0.0.1:38475_solr 
c:departments s:shard2  x:departments_shard2_replica1] o.a.s.c.CoreContainer 
Creating SolrCore 'departments_shard2_replica1' using configuration from 
collection departments
   [junit4]   2> 365805 INFO  (qtp1663009772-1164) [n:127.0.0.1:38475_solr 
c:departments s:shard2 r:core_node3 x:departments_shard2_replica1] 
o.a.s.c.SolrCore [[departments_shard2_replica1] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node1/departments_shard2_replica1],
 
dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node1/./departments_shard2_replica1/data/]
   [junit4]   2> 365806 INFO  (qtp444453979-1197) [n:127.0.0.1:34130_solr 
c:departments s:shard1  x:departments_shard1_replica1] o.a.s.c.CoreContainer 
Creating SolrCore 'departments_shard1_replica1' using configuration from 
collection departments
   [junit4]   2> 365806 INFO  (qtp444453979-1197) [n:127.0.0.1:34130_solr 
c:departments s:shard1 r:core_node4 x:departments_shard1_replica1] 
o.a.s.c.SolrCore [[departments_shard1_replica1] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node5/departments_shard1_replica1],
 
dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node5/./departments_shard1_replica1/data/]
   [junit4]   2> 365861 INFO  (qtp1231523479-1175) [n:127.0.0.1:54784_solr 
c:departments s:shard1 r:core_node2 x:departments_shard1_replica2] 
o.a.s.u.UpdateHandler Using UpdateLog implementation: 
org.apache.solr.update.UpdateLog
   [junit4]   2> 365861 INFO  (qtp1231523479-1175) [n:127.0.0.1:54784_solr 
c:departments s:shard1 r:core_node2 x:departments_shard1_replica2] 
o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 365862 INFO  (qtp1231523479-1175) [n:127.0.0.1:54784_solr 
c:departments s:shard1 r:core_node2 x:departments_shard1_replica2] 
o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 365862 INFO  (qtp1231523479-1175) [n:127.0.0.1:54784_solr 
c:departments s:shard1 r:core_node2 x:departments_shard1_replica2] 
o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 365870 INFO  (qtp557959012-1156) [n:127.0.0.1:39449_solr 
c:departments s:shard2 r:core_node1 x:departments_shard2_replica2] 
o.a.s.u.UpdateHandler Using UpdateLog implementation: 
org.apache.solr.update.UpdateLog
   [junit4]   2> 365870 INFO  (qtp557959012-1156) [n:127.0.0.1:39449_solr 
c:departments s:shard2 r:core_node1 x:departments_shard2_replica2] 
o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 365871 INFO  (qtp557959012-1156) [n:127.0.0.1:39449_solr 
c:departments s:shard2 r:core_node1 x:departments_shard2_replica2] 
o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 365871 INFO  (qtp557959012-1156) [n:127.0.0.1:39449_solr 
c:departments s:shard2 r:core_node1 x:departments_shard2_replica2] 
o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 365871 INFO  (qtp1231523479-1175) [n:127.0.0.1:54784_solr 
c:departments s:shard1 r:core_node2 x:departments_shard1_replica2] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@48e4f4ff[departments_shard1_replica2] main]
   [junit4]   2> 365872 INFO  (qtp557959012-1156) [n:127.0.0.1:39449_solr 
c:departments s:shard2 r:core_node1 x:departments_shard2_replica2] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@4a060658[departments_shard2_replica2] main]
   [junit4]   2> 365874 INFO  (qtp557959012-1156) [n:127.0.0.1:39449_solr 
c:departments s:shard2 r:core_node1 x:departments_shard2_replica2] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/solrCloudCollectionConfig
   [junit4]   2> 365874 INFO  (qtp557959012-1156) [n:127.0.0.1:39449_solr 
c:departments s:shard2 r:core_node1 x:departments_shard2_replica2] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/solrCloudCollectionConfig
   [junit4]   2> 365874 INFO  (qtp557959012-1156) [n:127.0.0.1:39449_solr 
c:departments s:shard2 r:core_node1 x:departments_shard2_replica2] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 365875 INFO  (qtp1231523479-1175) [n:127.0.0.1:54784_solr 
c:departments s:shard1 r:core_node2 x:departments_shard1_replica2] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/solrCloudCollectionConfig
   [junit4]   2> 365875 INFO  (qtp1231523479-1175) [n:127.0.0.1:54784_solr 
c:departments s:shard1 r:core_node2 x:departments_shard1_replica2] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/solrCloudCollectionConfig
   [junit4]   2> 365875 INFO  (qtp1231523479-1175) [n:127.0.0.1:54784_solr 
c:departments s:shard1 r:core_node2 x:departments_shard1_replica2] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 365950 INFO  (qtp1231523479-1175) [n:127.0.0.1:54784_solr 
c:departments s:shard1 r:core_node2 x:departments_shard1_replica2] 
o.a.s.u.UpdateLog Could not find max version in index or recent updates, using 
new clock 1547296866862891008
   [junit4]   2> 365952 INFO  (qtp557959012-1156) [n:127.0.0.1:39449_solr 
c:departments s:shard2 r:core_node1 x:departments_shard2_replica2] 
o.a.s.u.UpdateLog Could not find max version in index or recent updates, using 
new clock 1547296866864988160
   [junit4]   2> 365962 INFO  
(searcherExecutor-477-thread-1-processing-n:127.0.0.1:54784_solr 
x:departments_shard1_replica2 s:shard1 c:departments r:core_node2) 
[n:127.0.0.1:54784_solr c:departments s:shard1 r:core_node2 
x:departments_shard1_replica2] o.a.s.c.SolrCore [departments_shard1_replica2] 
Registered new searcher Searcher@48e4f4ff[departments_shard1_replica2] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 365976 INFO  (qtp557959012-1156) [n:127.0.0.1:39449_solr 
c:departments s:shard2 r:core_node1 x:departments_shard2_replica2] 
o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for 
shard shard2: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 365978 INFO  (qtp1231523479-1175) [n:127.0.0.1:54784_solr 
c:departments s:shard1 r:core_node2 x:departments_shard1_replica2] 
o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for 
shard shard1: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 365979 INFO  
(searcherExecutor-478-thread-1-processing-n:127.0.0.1:39449_solr 
x:departments_shard2_replica2 s:shard2 c:departments r:core_node1) 
[n:127.0.0.1:39449_solr c:departments s:shard2 r:core_node1 
x:departments_shard2_replica2] o.a.s.c.SolrCore [departments_shard2_replica2] 
Registered new searcher Searcher@4a060658[departments_shard2_replica2] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 366002 INFO  (qtp444453979-1197) [n:127.0.0.1:34130_solr 
c:departments s:shard1 r:core_node4 x:departments_shard1_replica1] 
o.a.s.u.UpdateHandler Using UpdateLog implementation: 
org.apache.solr.update.UpdateLog
   [junit4]   2> 366002 INFO  (qtp444453979-1197) [n:127.0.0.1:34130_solr 
c:departments s:shard1 r:core_node4 x:departments_shard1_replica1] 
o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 366003 INFO  (qtp444453979-1197) [n:127.0.0.1:34130_solr 
c:departm

[...truncated too long message...]

ate.json
   [junit4]   2>        at 
org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2>        at 
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2>        at 
org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
   [junit4]   2>        at 
org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:322)
   [junit4]   2>        at 
org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:319)
   [junit4]   2>        at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2>        at 
org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:319)
   [junit4]   2>        at 
org.apache.solr.cloud.overseer.ZkStateWriter.writePendingUpdates(ZkStateWriter.java:228)
   [junit4]   2>        at 
org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:225)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 377638 INFO  
(OverseerStateUpdate-96706052996988935-127.0.0.1:54784_solr-n_0000000000) 
[n:127.0.0.1:54784_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 
127.0.0.1:54784_solr
   [junit4]   2> 377641 INFO  
(zkCallback-171-thread-1-processing-n:127.0.0.1:38475_solr) 
[n:127.0.0.1:38475_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (5) -> (2)
   [junit4]   2> 377669 INFO  
(zkCallback-171-thread-3-processing-n:127.0.0.1:38475_solr) 
[n:127.0.0.1:38475_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (2) -> (1)
   [junit4]   2> 377760 INFO  
(OverseerStateUpdate-96706052996988937-127.0.0.1:38475_solr-n_0000000001) 
[n:127.0.0.1:38475_solr    ] o.a.s.c.Overseer According to ZK I 
(id=96706052996988937-127.0.0.1:38475_solr-n_0000000001) am no longer a leader.
   [junit4]   2> 377760 INFO  
(OverseerStateUpdate-96706052996988937-127.0.0.1:38475_solr-n_0000000001) 
[n:127.0.0.1:38475_solr    ] o.a.s.c.Overseer According to ZK I 
(id=96706052996988937-127.0.0.1:38475_solr-n_0000000001) am no longer a leader.
   [junit4]   2> 377760 INFO  
(OverseerStateUpdate-96706052996988937-127.0.0.1:38475_solr-n_0000000001) 
[n:127.0.0.1:38475_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 
127.0.0.1:38475_solr
   [junit4]   2> 377762 INFO  
(OverseerCollectionConfigSetProcessor-96706052996988937-127.0.0.1:38475_solr-n_0000000001)
 [n:127.0.0.1:38475_solr    ] o.a.s.c.OverseerTaskProcessor According to ZK I 
(id=96706052996988937-127.0.0.1:38475_solr-n_0000000001) am no longer a leader.
   [junit4]   2> 377762 ERROR 
(OverseerCollectionConfigSetProcessor-96706052996988937-127.0.0.1:38475_solr-n_0000000001)
 [n:127.0.0.1:38475_solr    ] o.a.s.c.OverseerTaskProcessor 
:org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode 
= Session expired for /overseer/collection-queue-work
   [junit4]   2>        at 
org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2>        at 
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2>        at 
org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2>        at 
org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:339)
   [junit4]   2>        at 
org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:336)
   [junit4]   2>        at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2>        at 
org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:336)
   [junit4]   2>        at 
org.apache.solr.cloud.DistributedQueue.fetchZkChildren(DistributedQueue.java:299)
   [junit4]   2>        at 
org.apache.solr.cloud.OverseerTaskQueue.getTailId(OverseerTaskQueue.java:264)
   [junit4]   2>        at 
org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:162)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 377763 INFO  
(OverseerCollectionConfigSetProcessor-96706052996988937-127.0.0.1:38475_solr-n_0000000001)
 [n:127.0.0.1:38475_solr    ] o.a.s.c.OverseerTaskProcessor According to ZK I 
(id=96706052996988937-127.0.0.1:38475_solr-n_0000000001) am no longer a leader.
   [junit4]   2> 377766 INFO  (jetty-launcher-151-thread-3) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@4cd97e8e{/solr,null,UNAVAILABLE}
   [junit4]   2> 377771 WARN  
(zkCallback-171-thread-3-processing-n:127.0.0.1:38475_solr) 
[n:127.0.0.1:38475_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, 
but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 377790 INFO  (jetty-launcher-151-thread-4) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@2e50d386{/solr,null,UNAVAILABLE}
   [junit4]   2> 378992 WARN  
(zkCallback-180-thread-1-processing-n:127.0.0.1:58449_solr) 
[n:127.0.0.1:58449_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, 
but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 378993 INFO  (jetty-launcher-151-thread-1) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@75f1e9bd{/solr,null,UNAVAILABLE}
   [junit4]   2> 379095 INFO  (jetty-launcher-151-thread-2) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@6a6770cc{/solr,null,UNAVAILABLE}
   [junit4]   2> 379098 INFO  
(SUITE-TestSubQueryTransformerDistrib-seed#[B6C73D6CCF57C41A]-worker) [    ] 
o.a.s.c.ZkTestServer connecting to 127.0.0.1:51643 51643
   [junit4]   2> 379206 INFO  (Thread-392) [    ] o.a.s.c.ZkTestServer 
connecting to 127.0.0.1:51643 51643
   [junit4]   2> 379207 WARN  (Thread-392) [    ] o.a.s.c.ZkTestServer Watch 
limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2>        6       /solr/aliases.json
   [junit4]   2>        6       /solr/clusterprops.json
   [junit4]   2>        5       /solr/security.json
   [junit4]   2>        5       /solr/configs/solrCloudCollectionConfig
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2>        6       /solr/clusterstate.json
   [junit4]   2>        4       /solr/collections/departments/state.json
   [junit4]   2>        4       /solr/collections/people/state.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2>        21      /solr/overseer/queue
   [junit4]   2>        21      /solr/overseer/collection-queue-work
   [junit4]   2>        8       /solr/overseer/queue-work
   [junit4]   2>        6       /solr/live_nodes
   [junit4]   2>        6       /solr/collections
   [junit4]   2> 
   [junit4]   2> 379207 INFO  
(SUITE-TestSubQueryTransformerDistrib-seed#[B6C73D6CCF57C41A]-worker) [    ] 
o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> Oct 04, 2016 9:41:35 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked 
thread(s).
   [junit4]   2> Oct 04, 2016 9:41:55 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at 
org.apache.solr.response.transform.TestSubQueryTransformerDistrib: 
   [junit4]   2>    1) Thread[id=1369, 
name=OverseerHdfsCoreFailoverThread-96706052996988937-127.0.0.1:38475_solr-n_0000000001,
 state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
   [junit4]   2>         at java.lang.Thread.sleep(Native Method)
   [junit4]   2>         at 
org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Oct 04, 2016 9:41:55 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=1369, 
name=OverseerHdfsCoreFailoverThread-96706052996988937-127.0.0.1:38475_solr-n_0000000001,
 state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
   [junit4]   2> Oct 04, 2016 9:41:55 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: All leaked threads terminated.
   [junit4]   2> NOTE: test params are: codec=CheapBastard, 
sim=ClassicSimilarity, locale=sv, timezone=Pacific/Norfolk
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 
1.8.0_102 (64-bit)/cpus=4,threads=1,free=69174440,total=368050176
   [junit4]   2> NOTE: All tests run in this JVM: [TestIndexingPerformance, 
CdcrUpdateLogTest, CurrencyFieldXmlFileTest, TestLeaderElectionZkExpiry, 
DeleteInactiveReplicaTest, TestPhraseSuggestions, TestExtendedDismaxParser, 
TestSolrQueryParser, TestRestoreCore, TestHdfsBackupRestoreCore, 
TestAnalyzeInfixSuggestions, TestJettySolrRunner, TestRandomFlRTGCloud, 
TestPerFieldSimilarityWithDefaultOverride, TestReload, QueryParsingTest, 
UpdateParamsTest, TestAuthorizationFramework, TestSubQueryTransformerDistrib]
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=TestSubQueryTransformerDistrib -Dtests.seed=B6C73D6CCF57C41A 
-Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=sv 
-Dtests.timezone=Pacific/Norfolk -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J2 | TestSubQueryTransformerDistrib (suite) <<<
   [junit4]    > Throwable #1: 
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE 
scope at org.apache.solr.response.transform.TestSubQueryTransformerDistrib: 
   [junit4]    >    1) Thread[id=1369, 
name=OverseerHdfsCoreFailoverThread-96706052996988937-127.0.0.1:38475_solr-n_0000000001,
 state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at 
org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([B6C73D6CCF57C41A]:0)
   [junit4] Completed [50/637 (1!)] on J2 in 51.56s, 1 test, 1 error <<< 
FAILURES!

[...truncated 62190 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Reply via email to