Build: https://builds.apache.org/job/Lucene-Solr-Tests-6.x/465/
1 tests failed. FAILED: junit.framework.TestSuite.org.apache.solr.response.transform.TestSubQueryTransformerDistrib Error Message: 1 thread leaked from SUITE scope at org.apache.solr.response.transform.TestSubQueryTransformerDistrib: 1) Thread[id=1369, name=OverseerHdfsCoreFailoverThread-96706052996988937-127.0.0.1:38475_solr-n_0000000001, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.] at java.lang.Thread.sleep(Native Method) at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137) at java.lang.Thread.run(Thread.java:745) Stack Trace: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.response.transform.TestSubQueryTransformerDistrib: 1) Thread[id=1369, name=OverseerHdfsCoreFailoverThread-96706052996988937-127.0.0.1:38475_solr-n_0000000001, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.] at java.lang.Thread.sleep(Native Method) at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137) at java.lang.Thread.run(Thread.java:745) at __randomizedtesting.SeedInfo.seed([B6C73D6CCF57C41A]:0) Build Log: [...truncated 10791 lines...] [junit4] Suite: org.apache.solr.response.transform.TestSubQueryTransformerDistrib [junit4] 2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/init-core-data-001 [junit4] 2> 347909 INFO (SUITE-TestSubQueryTransformerDistrib-seed#[B6C73D6CCF57C41A]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=None) [junit4] 2> 347910 INFO (SUITE-TestSubQueryTransformerDistrib-seed#[B6C73D6CCF57C41A]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 5 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001 [junit4] 2> 347910 INFO (SUITE-TestSubQueryTransformerDistrib-seed#[B6C73D6CCF57C41A]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER [junit4] 2> 347913 INFO (Thread-392) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0 [junit4] 2> 347914 INFO (Thread-392) [ ] o.a.s.c.ZkTestServer Starting server [junit4] 2> 348017 INFO (SUITE-TestSubQueryTransformerDistrib-seed#[B6C73D6CCF57C41A]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:51643 [junit4] 2> 348033 INFO (jetty-launcher-151-thread-1) [ ] o.e.j.s.Server jetty-9.3.8.v20160314 [junit4] 2> 348038 INFO (jetty-launcher-151-thread-5) [ ] o.e.j.s.Server jetty-9.3.8.v20160314 [junit4] 2> 348039 INFO (jetty-launcher-151-thread-4) [ ] o.e.j.s.Server jetty-9.3.8.v20160314 [junit4] 2> 348047 INFO (jetty-launcher-151-thread-2) [ ] o.e.j.s.Server jetty-9.3.8.v20160314 [junit4] 2> 348098 INFO (jetty-launcher-151-thread-5) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@22222ad1{/solr,null,AVAILABLE} [junit4] 2> 348099 INFO (jetty-launcher-151-thread-5) [ ] o.e.j.s.ServerConnector Started ServerConnector@4d016fb0{HTTP/1.1,[http/1.1]}{127.0.0.1:39449} [junit4] 2> 348099 INFO (jetty-launcher-151-thread-5) [ ] o.e.j.s.Server Started @356564ms [junit4] 2> 348099 INFO (jetty-launcher-151-thread-5) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=39449} [junit4] 2> 348099 INFO (jetty-launcher-151-thread-5) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 6.3.0 [junit4] 2> 348099 INFO (jetty-launcher-151-thread-5) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null [junit4] 2> 348099 INFO (jetty-launcher-151-thread-5) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null [junit4] 2> 348099 INFO (jetty-launcher-151-thread-5) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2016-10-04T21:41:03.932Z [junit4] 2> 348222 INFO (jetty-launcher-151-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2e50d386{/solr,null,AVAILABLE} [junit4] 2> 348222 INFO (jetty-launcher-151-thread-1) [ ] o.e.j.s.ServerConnector Started ServerConnector@74d6604d{HTTP/1.1,[http/1.1]}{127.0.0.1:38475} [junit4] 2> 348222 INFO (jetty-launcher-151-thread-1) [ ] o.e.j.s.Server Started @356688ms [junit4] 2> 348222 INFO (jetty-launcher-151-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=38475} [junit4] 2> 348223 INFO (jetty-launcher-151-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 6.3.0 [junit4] 2> 348223 INFO (jetty-launcher-151-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null [junit4] 2> 348223 INFO (jetty-launcher-151-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null [junit4] 2> 348223 INFO (jetty-launcher-151-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2016-10-04T21:41:04.056Z [junit4] 2> 348228 INFO (jetty-launcher-151-thread-3) [ ] o.e.j.s.Server jetty-9.3.8.v20160314 [junit4] 2> 348252 INFO (jetty-launcher-151-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4cd97e8e{/solr,null,AVAILABLE} [junit4] 2> 348253 INFO (jetty-launcher-151-thread-2) [ ] o.e.j.s.ServerConnector Started ServerConnector@f5426ec{HTTP/1.1,[http/1.1]}{127.0.0.1:54784} [junit4] 2> 348253 INFO (jetty-launcher-151-thread-2) [ ] o.e.j.s.Server Started @356718ms [junit4] 2> 348253 INFO (jetty-launcher-151-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=54784} [junit4] 2> 348253 INFO (jetty-launcher-151-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 6.3.0 [junit4] 2> 348253 INFO (jetty-launcher-151-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null [junit4] 2> 348253 INFO (jetty-launcher-151-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null [junit4] 2> 348253 INFO (jetty-launcher-151-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2016-10-04T21:41:04.086Z [junit4] 2> 348285 INFO (jetty-launcher-151-thread-4) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@75f1e9bd{/solr,null,AVAILABLE} [junit4] 2> 348285 INFO (jetty-launcher-151-thread-4) [ ] o.e.j.s.ServerConnector Started ServerConnector@1a7a90c7{HTTP/1.1,[http/1.1]}{127.0.0.1:58449} [junit4] 2> 348285 INFO (jetty-launcher-151-thread-4) [ ] o.e.j.s.Server Started @356751ms [junit4] 2> 348285 INFO (jetty-launcher-151-thread-4) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=58449} [junit4] 2> 348286 INFO (jetty-launcher-151-thread-4) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 6.3.0 [junit4] 2> 348286 INFO (jetty-launcher-151-thread-4) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null [junit4] 2> 348286 INFO (jetty-launcher-151-thread-4) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null [junit4] 2> 348286 INFO (jetty-launcher-151-thread-4) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2016-10-04T21:41:04.119Z [junit4] 2> 348338 INFO (jetty-launcher-151-thread-3) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6a6770cc{/solr,null,AVAILABLE} [junit4] 2> 348339 INFO (jetty-launcher-151-thread-3) [ ] o.e.j.s.ServerConnector Started ServerConnector@6b5c71b1{HTTP/1.1,[http/1.1]}{127.0.0.1:34130} [junit4] 2> 348339 INFO (jetty-launcher-151-thread-3) [ ] o.e.j.s.Server Started @356805ms [junit4] 2> 348339 INFO (jetty-launcher-151-thread-3) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=34130} [junit4] 2> 348339 INFO (jetty-launcher-151-thread-3) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 6.3.0 [junit4] 2> 348339 INFO (jetty-launcher-151-thread-3) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null [junit4] 2> 348339 INFO (jetty-launcher-151-thread-3) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null [junit4] 2> 348339 INFO (jetty-launcher-151-thread-3) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2016-10-04T21:41:04.172Z [junit4] 2> 348378 INFO (jetty-launcher-151-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading... [junit4] 2> 348391 INFO (jetty-launcher-151-thread-2) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true [junit4] 2> 348395 INFO (jetty-launcher-151-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51643/solr [junit4] 2> 348426 INFO (jetty-launcher-151-thread-3) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading... [junit4] 2> 348496 INFO (jetty-launcher-151-thread-5) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading... [junit4] 2> 348494 INFO (jetty-launcher-151-thread-4) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading... [junit4] 2> 348434 INFO (jetty-launcher-151-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading... [junit4] 2> 348626 INFO (jetty-launcher-151-thread-5) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true [junit4] 2> 348626 INFO (jetty-launcher-151-thread-5) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51643/solr [junit4] 2> 348630 INFO (jetty-launcher-151-thread-4) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true [junit4] 2> 348646 INFO (jetty-launcher-151-thread-1) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true [junit4] 2> 348649 INFO (jetty-launcher-151-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51643/solr [junit4] 2> 348663 INFO (jetty-launcher-151-thread-2) [n:127.0.0.1:54784_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:54784_solr [junit4] 2> 348688 INFO (jetty-launcher-151-thread-2) [n:127.0.0.1:54784_solr ] o.a.s.c.Overseer Overseer (id=96706052996988935-127.0.0.1:54784_solr-n_0000000000) starting [junit4] 2> 348694 INFO (jetty-launcher-151-thread-1) [n:127.0.0.1:38475_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38475_solr [junit4] 2> 348696 INFO (jetty-launcher-151-thread-4) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51643/solr [junit4] 2> 348759 INFO (jetty-launcher-151-thread-3) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true [junit4] 2> 348794 INFO (zkCallback-161-thread-1-processing-n:127.0.0.1:54784_solr) [n:127.0.0.1:54784_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) [junit4] 2> 348896 INFO (zkCallback-171-thread-1-processing-n:127.0.0.1:38475_solr) [n:127.0.0.1:38475_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) [junit4] 2> 348900 INFO (jetty-launcher-151-thread-1) [n:127.0.0.1:38475_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node1/. [junit4] 2> 348901 INFO (jetty-launcher-151-thread-3) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51643/solr [junit4] 2> 349199 INFO (jetty-launcher-151-thread-2) [n:127.0.0.1:54784_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54784_solr [junit4] 2> 349257 INFO (jetty-launcher-151-thread-5) [n:127.0.0.1:39449_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) [junit4] 2> 349554 INFO (zkCallback-171-thread-1-processing-n:127.0.0.1:38475_solr) [n:127.0.0.1:38475_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2) [junit4] 2> 349555 INFO (zkCallback-161-thread-1-processing-n:127.0.0.1:54784_solr) [n:127.0.0.1:54784_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2) [junit4] 2> 349592 INFO (jetty-launcher-151-thread-3) [n:127.0.0.1:34130_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2) [junit4] 2> 349598 INFO (jetty-launcher-151-thread-5) [n:127.0.0.1:39449_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39449_solr [junit4] 2> 349621 INFO (zkCallback-171-thread-1-processing-n:127.0.0.1:38475_solr) [n:127.0.0.1:38475_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3) [junit4] 2> 349622 INFO (jetty-launcher-151-thread-4) [n:127.0.0.1:58449_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3) [junit4] 2> 349623 INFO (jetty-launcher-151-thread-3) [n:127.0.0.1:34130_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34130_solr [junit4] 2> 349625 INFO (zkCallback-161-thread-1-processing-n:127.0.0.1:54784_solr) [n:127.0.0.1:54784_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3) [junit4] 2> 349627 INFO (zkCallback-177-thread-1-processing-n:127.0.0.1:39449_solr) [n:127.0.0.1:39449_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2) [junit4] 2> 349662 INFO (zkCallback-177-thread-2-processing-n:127.0.0.1:39449_solr) [n:127.0.0.1:39449_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (4) [junit4] 2> 349667 INFO (jetty-launcher-151-thread-2) [n:127.0.0.1:54784_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node2/. [junit4] 2> 349672 INFO (jetty-launcher-151-thread-4) [n:127.0.0.1:58449_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58449_solr [junit4] 2> 349673 INFO (zkCallback-171-thread-1-processing-n:127.0.0.1:38475_solr) [n:127.0.0.1:38475_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4) [junit4] 2> 349680 INFO (jetty-launcher-151-thread-5) [n:127.0.0.1:39449_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node3/. [junit4] 2> 349684 INFO (jetty-launcher-151-thread-3) [n:127.0.0.1:34130_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node5/. [junit4] 2> 349686 INFO (zkCallback-171-thread-1-processing-n:127.0.0.1:38475_solr) [n:127.0.0.1:38475_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5) [junit4] 2> 349689 INFO (zkCallback-177-thread-2-processing-n:127.0.0.1:39449_solr) [n:127.0.0.1:39449_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5) [junit4] 2> 349690 INFO (zkCallback-161-thread-2-processing-n:127.0.0.1:54784_solr) [n:127.0.0.1:54784_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (5) [junit4] 2> 349734 INFO (zkCallback-180-thread-1-processing-n:127.0.0.1:58449_solr) [n:127.0.0.1:58449_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (5) [junit4] 2> 349734 INFO (zkCallback-178-thread-1-processing-n:127.0.0.1:34130_solr) [n:127.0.0.1:34130_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (5) [junit4] 2> 349791 INFO (jetty-launcher-151-thread-4) [n:127.0.0.1:58449_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node4/. [junit4] 2> 350011 INFO (SUITE-TestSubQueryTransformerDistrib-seed#[B6C73D6CCF57C41A]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (5) [junit4] 2> 352073 INFO (qtp557959012-1153) [n:127.0.0.1:39449_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&property.schema=schema-docValuesJoin.xml&property.config=solrconfig-doctransformers.xml&collection.configName=solrCloudCollectionConfig&name=people&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true [junit4] 2> 352087 INFO (OverseerThreadFactory-448-thread-1-processing-n:127.0.0.1:54784_solr) [n:127.0.0.1:54784_solr ] o.a.s.c.CreateCollectionCmd Create collection people [junit4] 2> 352263 INFO (qtp557959012-1151) [n:127.0.0.1:39449_solr ] o.a.s.h.a.CoreAdminOperation core create command property.schema=schema-docValuesJoin.xml&property.config=solrconfig-doctransformers.xml&qt=/admin/cores&collection.configName=solrCloudCollectionConfig&name=people_shard2_replica2&action=CREATE&numShards=2&collection=people&shard=shard2&wt=javabin&version=2 [junit4] 2> 352290 INFO (qtp1201973810-1191) [n:127.0.0.1:58449_solr ] o.a.s.h.a.CoreAdminOperation core create command property.schema=schema-docValuesJoin.xml&property.config=solrconfig-doctransformers.xml&qt=/admin/cores&collection.configName=solrCloudCollectionConfig&name=people_shard1_replica1&action=CREATE&numShards=2&collection=people&shard=shard1&wt=javabin&version=2 [junit4] 2> 352291 INFO (qtp1231523479-1176) [n:127.0.0.1:54784_solr ] o.a.s.h.a.CoreAdminOperation core create command property.schema=schema-docValuesJoin.xml&property.config=solrconfig-doctransformers.xml&qt=/admin/cores&collection.configName=solrCloudCollectionConfig&name=people_shard2_replica1&action=CREATE&numShards=2&collection=people&shard=shard2&wt=javabin&version=2 [junit4] 2> 352292 INFO (qtp1663009772-1163) [n:127.0.0.1:38475_solr ] o.a.s.h.a.CoreAdminOperation core create command property.schema=schema-docValuesJoin.xml&property.config=solrconfig-doctransformers.xml&qt=/admin/cores&collection.configName=solrCloudCollectionConfig&name=people_shard1_replica2&action=CREATE&numShards=2&collection=people&shard=shard1&wt=javabin&version=2 [junit4] 2> 352413 INFO (zkCallback-180-thread-1-processing-n:127.0.0.1:58449_solr) [n:127.0.0.1:58449_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5]) [junit4] 2> 352413 INFO (zkCallback-171-thread-1-processing-n:127.0.0.1:38475_solr) [n:127.0.0.1:38475_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5]) [junit4] 2> 352413 INFO (zkCallback-177-thread-2-processing-n:127.0.0.1:39449_solr) [n:127.0.0.1:39449_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5]) [junit4] 2> 352413 INFO (zkCallback-161-thread-1-processing-n:127.0.0.1:54784_solr) [n:127.0.0.1:54784_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5]) [junit4] 2> 353345 INFO (qtp1663009772-1163) [n:127.0.0.1:38475_solr c:people s:shard1 x:people_shard1_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.3.0 [junit4] 2> 353348 INFO (qtp1201973810-1191) [n:127.0.0.1:58449_solr c:people s:shard1 x:people_shard1_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.3.0 [junit4] 2> 353355 INFO (qtp1231523479-1176) [n:127.0.0.1:54784_solr c:people s:shard2 x:people_shard2_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.3.0 [junit4] 2> 353362 INFO (qtp557959012-1151) [n:127.0.0.1:39449_solr c:people s:shard2 x:people_shard2_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.3.0 [junit4] 2> 353376 INFO (qtp1663009772-1163) [n:127.0.0.1:38475_solr c:people s:shard1 x:people_shard1_replica2] o.a.s.s.IndexSchema [people_shard1_replica2] Schema name=doc-values-for-Join [junit4] 2> 353398 INFO (qtp1201973810-1191) [n:127.0.0.1:58449_solr c:people s:shard1 x:people_shard1_replica1] o.a.s.s.IndexSchema [people_shard1_replica1] Schema name=doc-values-for-Join [junit4] 2> 353472 WARN (qtp1663009772-1163) [n:127.0.0.1:38475_solr c:people s:shard1 x:people_shard1_replica2] o.a.s.s.IndexSchema [people_shard1_replica2] default search field in schema is id. WARNING: Deprecated, please use 'df' on request instead. [junit4] 2> 353473 INFO (qtp1663009772-1163) [n:127.0.0.1:38475_solr c:people s:shard1 x:people_shard1_replica2] o.a.s.s.IndexSchema Loaded schema doc-values-for-Join/1.6 with uniqueid field id [junit4] 2> 353480 INFO (qtp557959012-1151) [n:127.0.0.1:39449_solr c:people s:shard2 x:people_shard2_replica2] o.a.s.s.IndexSchema [people_shard2_replica2] Schema name=doc-values-for-Join [junit4] 2> 353483 INFO (qtp1231523479-1176) [n:127.0.0.1:54784_solr c:people s:shard2 x:people_shard2_replica1] o.a.s.s.IndexSchema [people_shard2_replica1] Schema name=doc-values-for-Join [junit4] 2> 353497 WARN (qtp1201973810-1191) [n:127.0.0.1:58449_solr c:people s:shard1 x:people_shard1_replica1] o.a.s.s.IndexSchema [people_shard1_replica1] default search field in schema is id. WARNING: Deprecated, please use 'df' on request instead. [junit4] 2> 353531 INFO (qtp1201973810-1191) [n:127.0.0.1:58449_solr c:people s:shard1 x:people_shard1_replica1] o.a.s.s.IndexSchema Loaded schema doc-values-for-Join/1.6 with uniqueid field id [junit4] 2> 353530 WARN (qtp1231523479-1176) [n:127.0.0.1:54784_solr c:people s:shard2 x:people_shard2_replica1] o.a.s.s.IndexSchema [people_shard2_replica1] default search field in schema is id. WARNING: Deprecated, please use 'df' on request instead. [junit4] 2> 353532 INFO (qtp1663009772-1163) [n:127.0.0.1:38475_solr c:people s:shard1 x:people_shard1_replica2] o.a.s.c.CoreContainer Creating SolrCore 'people_shard1_replica2' using configuration from collection people [junit4] 2> 353533 INFO (qtp1663009772-1163) [n:127.0.0.1:38475_solr c:people s:shard1 r:core_node2 x:people_shard1_replica2] o.a.s.c.SolrCore [[people_shard1_replica2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node1/people_shard1_replica2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node1/./people_shard1_replica2/data/] [junit4] 2> 353500 WARN (qtp557959012-1151) [n:127.0.0.1:39449_solr c:people s:shard2 x:people_shard2_replica2] o.a.s.s.IndexSchema [people_shard2_replica2] default search field in schema is id. WARNING: Deprecated, please use 'df' on request instead. [junit4] 2> 353538 INFO (qtp1201973810-1191) [n:127.0.0.1:58449_solr c:people s:shard1 x:people_shard1_replica1] o.a.s.c.CoreContainer Creating SolrCore 'people_shard1_replica1' using configuration from collection people [junit4] 2> 353538 INFO (qtp1201973810-1191) [n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.c.SolrCore [[people_shard1_replica1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node4/people_shard1_replica1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node4/./people_shard1_replica1/data/] [junit4] 2> 353540 INFO (qtp1231523479-1176) [n:127.0.0.1:54784_solr c:people s:shard2 x:people_shard2_replica1] o.a.s.s.IndexSchema Loaded schema doc-values-for-Join/1.6 with uniqueid field id [junit4] 2> 353540 INFO (qtp557959012-1151) [n:127.0.0.1:39449_solr c:people s:shard2 x:people_shard2_replica2] o.a.s.s.IndexSchema Loaded schema doc-values-for-Join/1.6 with uniqueid field id [junit4] 2> 353547 INFO (qtp557959012-1151) [n:127.0.0.1:39449_solr c:people s:shard2 x:people_shard2_replica2] o.a.s.c.CoreContainer Creating SolrCore 'people_shard2_replica2' using configuration from collection people [junit4] 2> 353548 INFO (qtp557959012-1151) [n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.c.SolrCore [[people_shard2_replica2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node3/people_shard2_replica2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node3/./people_shard2_replica2/data/] [junit4] 2> 353549 INFO (qtp1231523479-1176) [n:127.0.0.1:54784_solr c:people s:shard2 x:people_shard2_replica1] o.a.s.c.CoreContainer Creating SolrCore 'people_shard2_replica1' using configuration from collection people [junit4] 2> 353549 INFO (qtp1231523479-1176) [n:127.0.0.1:54784_solr c:people s:shard2 r:core_node4 x:people_shard2_replica1] o.a.s.c.SolrCore [[people_shard2_replica1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node2/people_shard2_replica1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node2/./people_shard2_replica1/data/] [junit4] 2> 353694 INFO (qtp1663009772-1163) [n:127.0.0.1:38475_solr c:people s:shard1 r:core_node2 x:people_shard1_replica2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog [junit4] 2> 353695 INFO (qtp1663009772-1163) [n:127.0.0.1:38475_solr c:people s:shard1 r:core_node2 x:people_shard1_replica2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 [junit4] 2> 353696 INFO (qtp1663009772-1163) [n:127.0.0.1:38475_solr c:people s:shard1 r:core_node2 x:people_shard1_replica2] o.a.s.u.CommitTracker Hard AutoCommit: disabled [junit4] 2> 353696 INFO (qtp1663009772-1163) [n:127.0.0.1:38475_solr c:people s:shard1 r:core_node2 x:people_shard1_replica2] o.a.s.u.CommitTracker Soft AutoCommit: disabled [junit4] 2> 353697 INFO (qtp1663009772-1163) [n:127.0.0.1:38475_solr c:people s:shard1 r:core_node2 x:people_shard1_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@3a47d050[people_shard1_replica2] main] [junit4] 2> 353699 INFO (qtp1663009772-1163) [n:127.0.0.1:38475_solr c:people s:shard1 r:core_node2 x:people_shard1_replica2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/solrCloudCollectionConfig [junit4] 2> 353699 INFO (qtp1663009772-1163) [n:127.0.0.1:38475_solr c:people s:shard1 r:core_node2 x:people_shard1_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/solrCloudCollectionConfig [junit4] 2> 353699 INFO (qtp1663009772-1163) [n:127.0.0.1:38475_solr c:people s:shard1 r:core_node2 x:people_shard1_replica2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 [junit4] 2> 353737 INFO (qtp1231523479-1176) [n:127.0.0.1:54784_solr c:people s:shard2 r:core_node4 x:people_shard2_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog [junit4] 2> 353738 INFO (qtp1231523479-1176) [n:127.0.0.1:54784_solr c:people s:shard2 r:core_node4 x:people_shard2_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 [junit4] 2> 353738 INFO (qtp1231523479-1176) [n:127.0.0.1:54784_solr c:people s:shard2 r:core_node4 x:people_shard2_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled [junit4] 2> 353739 INFO (qtp1231523479-1176) [n:127.0.0.1:54784_solr c:people s:shard2 r:core_node4 x:people_shard2_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled [junit4] 2> 353739 INFO (qtp1663009772-1163) [n:127.0.0.1:38475_solr c:people s:shard1 r:core_node2 x:people_shard1_replica2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1547296854058729472 [junit4] 2> 353741 INFO (searcherExecutor-457-thread-1-processing-n:127.0.0.1:38475_solr x:people_shard1_replica2 s:shard1 c:people r:core_node2) [n:127.0.0.1:38475_solr c:people s:shard1 r:core_node2 x:people_shard1_replica2] o.a.s.c.SolrCore [people_shard1_replica2] Registered new searcher Searcher@3a47d050[people_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())} [junit4] 2> 353753 INFO (qtp1663009772-1163) [n:127.0.0.1:38475_solr c:people s:shard1 r:core_node2 x:people_shard1_replica2] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=9999ms [junit4] 2> 353766 INFO (qtp1201973810-1191) [n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog [junit4] 2> 353766 INFO (qtp1201973810-1191) [n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 [junit4] 2> 353767 INFO (qtp1201973810-1191) [n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled [junit4] 2> 353767 INFO (qtp1201973810-1191) [n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled [junit4] 2> 353767 INFO (qtp1201973810-1191) [n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@b7bf2ee[people_shard1_replica1] main] [junit4] 2> 353769 INFO (qtp1201973810-1191) [n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/solrCloudCollectionConfig [junit4] 2> 353769 INFO (qtp1201973810-1191) [n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/solrCloudCollectionConfig [junit4] 2> 353769 INFO (qtp1201973810-1191) [n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 [junit4] 2> 353769 INFO (qtp1201973810-1191) [n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1547296854090186752 [junit4] 2> 353774 INFO (searcherExecutor-458-thread-1-processing-n:127.0.0.1:58449_solr x:people_shard1_replica1 s:shard1 c:people r:core_node3) [n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.c.SolrCore [people_shard1_replica1] Registered new searcher Searcher@b7bf2ee[people_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())} [junit4] 2> 353775 INFO (qtp1231523479-1176) [n:127.0.0.1:54784_solr c:people s:shard2 r:core_node4 x:people_shard2_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@185ce4c0[people_shard2_replica1] main] [junit4] 2> 353776 INFO (qtp1231523479-1176) [n:127.0.0.1:54784_solr c:people s:shard2 r:core_node4 x:people_shard2_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/solrCloudCollectionConfig [junit4] 2> 353776 INFO (qtp1231523479-1176) [n:127.0.0.1:54784_solr c:people s:shard2 r:core_node4 x:people_shard2_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/solrCloudCollectionConfig [junit4] 2> 353776 INFO (qtp1231523479-1176) [n:127.0.0.1:54784_solr c:people s:shard2 r:core_node4 x:people_shard2_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 [junit4] 2> 353848 INFO (qtp557959012-1151) [n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog [junit4] 2> 353848 INFO (qtp557959012-1151) [n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 [junit4] 2> 353849 INFO (qtp557959012-1151) [n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.u.CommitTracker Hard AutoCommit: disabled [junit4] 2> 353849 INFO (qtp557959012-1151) [n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.u.CommitTracker Soft AutoCommit: disabled [junit4] 2> 353849 INFO (qtp1231523479-1176) [n:127.0.0.1:54784_solr c:people s:shard2 r:core_node4 x:people_shard2_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1547296854174072832 [junit4] 2> 353849 INFO (searcherExecutor-460-thread-1-processing-n:127.0.0.1:54784_solr x:people_shard2_replica1 s:shard2 c:people r:core_node4) [n:127.0.0.1:54784_solr c:people s:shard2 r:core_node4 x:people_shard2_replica1] o.a.s.c.SolrCore [people_shard2_replica1] Registered new searcher Searcher@185ce4c0[people_shard2_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())} [junit4] 2> 353859 INFO (qtp1231523479-1176) [n:127.0.0.1:54784_solr c:people s:shard2 r:core_node4 x:people_shard2_replica1] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoutin=9999ms [junit4] 2> 353859 INFO (qtp557959012-1151) [n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@324d3dc3[people_shard2_replica2] main] [junit4] 2> 353860 INFO (qtp557959012-1151) [n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/solrCloudCollectionConfig [junit4] 2> 353861 INFO (qtp557959012-1151) [n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/solrCloudCollectionConfig [junit4] 2> 353861 INFO (qtp557959012-1151) [n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 [junit4] 2> 353861 INFO (qtp557959012-1151) [n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1547296854186655744 [junit4] 2> 353870 INFO (searcherExecutor-459-thread-1-processing-n:127.0.0.1:39449_solr x:people_shard2_replica2 s:shard2 c:people r:core_node1) [n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.c.SolrCore [people_shard2_replica2] Registered new searcher Searcher@324d3dc3[people_shard2_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())} [junit4] 2> 353960 INFO (zkCallback-177-thread-2-processing-n:127.0.0.1:39449_solr) [n:127.0.0.1:39449_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5]) [junit4] 2> 353960 INFO (zkCallback-180-thread-1-processing-n:127.0.0.1:58449_solr) [n:127.0.0.1:58449_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5]) [junit4] 2> 353960 INFO (zkCallback-171-thread-1-processing-n:127.0.0.1:38475_solr) [n:127.0.0.1:38475_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5]) [junit4] 2> 353961 INFO (zkCallback-161-thread-2-processing-n:127.0.0.1:54784_solr) [n:127.0.0.1:54784_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5]) [junit4] 2> 354263 INFO (qtp1663009772-1163) [n:127.0.0.1:38475_solr c:people s:shard1 r:core_node2 x:people_shard1_replica2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. [junit4] 2> 354263 INFO (qtp1663009772-1163) [n:127.0.0.1:38475_solr c:people s:shard1 r:core_node2 x:people_shard1_replica2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync [junit4] 2> 354263 INFO (qtp1663009772-1163) [n:127.0.0.1:38475_solr c:people s:shard1 r:core_node2 x:people_shard1_replica2] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:38475/solr/people_shard1_replica2/ [junit4] 2> 354263 INFO (qtp1663009772-1163) [n:127.0.0.1:38475_solr c:people s:shard1 r:core_node2 x:people_shard1_replica2] o.a.s.u.PeerSync PeerSync: core=people_shard1_replica2 url=http://127.0.0.1:38475/solr START replicas=[http://127.0.0.1:58449/solr/people_shard1_replica1/] nUpdates=100 [junit4] 2> 354265 INFO (qtp1663009772-1163) [n:127.0.0.1:38475_solr c:people s:shard1 r:core_node2 x:people_shard1_replica2] o.a.s.u.PeerSync PeerSync: core=people_shard1_replica2 url=http://127.0.0.1:38475/solr DONE. We have no versions. sync failed. [junit4] 2> 354311 INFO (qtp1201973810-1187) [n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.c.S.Request [people_shard1_replica1] webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=1 [junit4] 2> 354312 INFO (qtp1663009772-1163) [n:127.0.0.1:38475_solr c:people s:shard1 r:core_node2 x:people_shard1_replica2] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate [junit4] 2> 354312 INFO (qtp1663009772-1163) [n:127.0.0.1:38475_solr c:people s:shard1 r:core_node2 x:people_shard1_replica2] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway [junit4] 2> 354339 INFO (qtp1663009772-1163) [n:127.0.0.1:38475_solr c:people s:shard1 r:core_node2 x:people_shard1_replica2] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:38475/solr/people_shard1_replica2/ shard1 [junit4] 2> 354362 INFO (qtp1231523479-1176) [n:127.0.0.1:54784_solr c:people s:shard2 r:core_node4 x:people_shard2_replica1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. [junit4] 2> 354362 INFO (qtp1231523479-1176) [n:127.0.0.1:54784_solr c:people s:shard2 r:core_node4 x:people_shard2_replica1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync [junit4] 2> 354362 INFO (qtp1231523479-1176) [n:127.0.0.1:54784_solr c:people s:shard2 r:core_node4 x:people_shard2_replica1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:54784/solr/people_shard2_replica1/ [junit4] 2> 354362 INFO (qtp1231523479-1176) [n:127.0.0.1:54784_solr c:people s:shard2 r:core_node4 x:people_shard2_replica1] o.a.s.u.PeerSync PeerSync: core=people_shard2_replica1 url=http://127.0.0.1:54784/solr START replicas=[http://127.0.0.1:39449/solr/people_shard2_replica2/] nUpdates=100 [junit4] 2> 354362 INFO (qtp1231523479-1176) [n:127.0.0.1:54784_solr c:people s:shard2 r:core_node4 x:people_shard2_replica1] o.a.s.u.PeerSync PeerSync: core=people_shard2_replica1 url=http://127.0.0.1:54784/solr DONE. We have no versions. sync failed. [junit4] 2> 354374 INFO (qtp557959012-1150) [n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.c.S.Request [people_shard2_replica2] webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=8 [junit4] 2> 354374 INFO (qtp1231523479-1176) [n:127.0.0.1:54784_solr c:people s:shard2 r:core_node4 x:people_shard2_replica1] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate [junit4] 2> 354374 INFO (qtp1231523479-1176) [n:127.0.0.1:54784_solr c:people s:shard2 r:core_node4 x:people_shard2_replica1] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway [junit4] 2> 354387 INFO (qtp1231523479-1176) [n:127.0.0.1:54784_solr c:people s:shard2 r:core_node4 x:people_shard2_replica1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:54784/solr/people_shard2_replica1/ shard2 [junit4] 2> 354493 INFO (zkCallback-177-thread-2-processing-n:127.0.0.1:39449_solr) [n:127.0.0.1:39449_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5]) [junit4] 2> 354493 INFO (zkCallback-171-thread-1-processing-n:127.0.0.1:38475_solr) [n:127.0.0.1:38475_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5]) [junit4] 2> 354493 INFO (zkCallback-180-thread-1-processing-n:127.0.0.1:58449_solr) [n:127.0.0.1:58449_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5]) [junit4] 2> 354494 INFO (zkCallback-161-thread-1-processing-n:127.0.0.1:54784_solr) [n:127.0.0.1:54784_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5]) [junit4] 2> 354495 INFO (qtp1663009772-1163) [n:127.0.0.1:38475_solr c:people s:shard1 r:core_node2 x:people_shard1_replica2] o.a.s.c.ZkController I am the leader, no recovery necessary [junit4] 2> 354502 INFO (qtp1663009772-1163) [n:127.0.0.1:38475_solr c:people s:shard1 r:core_node2 x:people_shard1_replica2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={property.schema=schema-docValuesJoin.xml&property.config=solrconfig-doctransformers.xml&qt=/admin/cores&collection.configName=solrCloudCollectionConfig&name=people_shard1_replica2&action=CREATE&numShards=2&collection=people&shard=shard1&wt=javabin&version=2} status=0 QTime=2210 [junit4] 2> 354545 INFO (qtp1231523479-1176) [n:127.0.0.1:54784_solr c:people s:shard2 r:core_node4 x:people_shard2_replica1] o.a.s.c.ZkController I am the leader, no recovery necessary [junit4] 2> 354551 INFO (qtp1231523479-1176) [n:127.0.0.1:54784_solr c:people s:shard2 r:core_node4 x:people_shard2_replica1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={property.schema=schema-docValuesJoin.xml&property.config=solrconfig-doctransformers.xml&qt=/admin/cores&collection.configName=solrCloudCollectionConfig&name=people_shard2_replica1&action=CREATE&numShards=2&collection=people&shard=shard2&wt=javabin&version=2} status=0 QTime=2259 [junit4] 2> 354695 INFO (zkCallback-177-thread-2-processing-n:127.0.0.1:39449_solr) [n:127.0.0.1:39449_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5]) [junit4] 2> 354695 INFO (zkCallback-171-thread-1-processing-n:127.0.0.1:38475_solr) [n:127.0.0.1:38475_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5]) [junit4] 2> 354696 INFO (zkCallback-180-thread-1-processing-n:127.0.0.1:58449_solr) [n:127.0.0.1:58449_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5]) [junit4] 2> 354697 INFO (zkCallback-161-thread-1-processing-n:127.0.0.1:54784_solr) [n:127.0.0.1:54784_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5]) [junit4] 2> 354778 INFO (qtp1201973810-1191) [n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.c.ZkController Core needs to recover:people_shard1_replica1 [junit4] 2> 354779 INFO (qtp1201973810-1191) [n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={property.schema=schema-docValuesJoin.xml&property.config=solrconfig-doctransformers.xml&qt=/admin/cores&collection.configName=solrCloudCollectionConfig&name=people_shard1_replica1&action=CREATE&numShards=2&collection=people&shard=shard1&wt=javabin&version=2} status=0 QTime=2489 [junit4] 2> 354780 INFO (updateExecutor-163-thread-1-processing-n:127.0.0.1:58449_solr x:people_shard1_replica1 s:shard1 c:people r:core_node3) [n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.u.DefaultSolrCoreState Running recovery [junit4] 2> 354785 INFO (recoveryExecutor-164-thread-1-processing-n:127.0.0.1:58449_solr x:people_shard1_replica1 s:shard1 c:people r:core_node3) [n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true [junit4] 2> 354786 INFO (recoveryExecutor-164-thread-1-processing-n:127.0.0.1:58449_solr x:people_shard1_replica1 s:shard1 c:people r:core_node3) [n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]] [junit4] 2> 354786 INFO (recoveryExecutor-164-thread-1-processing-n:127.0.0.1:58449_solr x:people_shard1_replica1 s:shard1 c:people r:core_node3) [n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[people_shard1_replica1] [junit4] 2> 354786 INFO (recoveryExecutor-164-thread-1-processing-n:127.0.0.1:58449_solr x:people_shard1_replica1 s:shard1 c:people r:core_node3) [n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null} [junit4] 2> 354786 INFO (recoveryExecutor-164-thread-1-processing-n:127.0.0.1:58449_solr x:people_shard1_replica1 s:shard1 c:people r:core_node3) [n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.c.RecoveryStrategy Publishing state of core [people_shard1_replica1] as recovering, leader is [http://127.0.0.1:38475/solr/people_shard1_replica2/] and I am [http://127.0.0.1:58449/solr/people_shard1_replica1/] [junit4] 2> 354797 INFO (recoveryExecutor-164-thread-1-processing-n:127.0.0.1:58449_solr x:people_shard1_replica1 s:shard1 c:people r:core_node3) [n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [http://127.0.0.1:38475/solr]; [WaitForState: action=PREPRECOVERY&core=people_shard1_replica2&nodeName=127.0.0.1:58449_solr&coreNodeName=core_node3&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true] [junit4] 2> 354806 INFO (qtp1663009772-1161) [n:127.0.0.1:38475_solr ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true [junit4] 2> 354822 INFO (qtp1663009772-1161) [n:127.0.0.1:38475_solr ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see people_shard1_replica2 (shard1 of people) have state: recovering [junit4] 2> 354822 INFO (qtp1663009772-1161) [n:127.0.0.1:38475_solr ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=people, shard=shard1, thisCore=people_shard1_replica2, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:58449_solr, coreNodeName=core_node3, onlyIfActiveCheckResult=false, nodeProps: core_node3:{"core":"people_shard1_replica1","base_url":"http://127.0.0.1:58449/solr","node_name":"127.0.0.1:58449_solr","state":"down"} [junit4] 2> 354870 INFO (qtp557959012-1151) [n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.c.ZkController Core needs to recover:people_shard2_replica2 [junit4] 2> 354871 INFO (qtp557959012-1151) [n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={property.schema=schema-docValuesJoin.xml&property.config=solrconfig-doctransformers.xml&qt=/admin/cores&collection.configName=solrCloudCollectionConfig&name=people_shard2_replica2&action=CREATE&numShards=2&collection=people&shard=shard2&wt=javabin&version=2} status=0 QTime=2608 [junit4] 2> 354905 INFO (updateExecutor-165-thread-1-processing-n:127.0.0.1:39449_solr x:people_shard2_replica2 s:shard2 c:people r:core_node1) [n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.u.DefaultSolrCoreState Running recovery [junit4] 2> 354906 INFO (zkCallback-171-thread-1-processing-n:127.0.0.1:38475_solr) [n:127.0.0.1:38475_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5]) [junit4] 2> 354906 INFO (zkCallback-180-thread-1-processing-n:127.0.0.1:58449_solr) [n:127.0.0.1:58449_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5]) [junit4] 2> 354907 INFO (zkCallback-177-thread-2-processing-n:127.0.0.1:39449_solr) [n:127.0.0.1:39449_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5]) [junit4] 2> 354908 INFO (zkCallback-161-thread-2-processing-n:127.0.0.1:54784_solr) [n:127.0.0.1:54784_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5]) [junit4] 2> 354912 INFO (qtp557959012-1153) [n:127.0.0.1:39449_solr ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas [junit4] 2> 354917 INFO (recoveryExecutor-166-thread-1-processing-n:127.0.0.1:39449_solr x:people_shard2_replica2 s:shard2 c:people r:core_node1) [n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true [junit4] 2> 354918 INFO (recoveryExecutor-166-thread-1-processing-n:127.0.0.1:39449_solr x:people_shard2_replica2 s:shard2 c:people r:core_node1) [n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]] [junit4] 2> 354918 INFO (recoveryExecutor-166-thread-1-processing-n:127.0.0.1:39449_solr x:people_shard2_replica2 s:shard2 c:people r:core_node1) [n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[people_shard2_replica2] [junit4] 2> 354918 INFO (recoveryExecutor-166-thread-1-processing-n:127.0.0.1:39449_solr x:people_shard2_replica2 s:shard2 c:people r:core_node1) [n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null} [junit4] 2> 354918 INFO (recoveryExecutor-166-thread-1-processing-n:127.0.0.1:39449_solr x:people_shard2_replica2 s:shard2 c:people r:core_node1) [n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.c.RecoveryStrategy Publishing state of core [people_shard2_replica2] as recovering, leader is [http://127.0.0.1:54784/solr/people_shard2_replica1/] and I am [http://127.0.0.1:39449/solr/people_shard2_replica2/] [junit4] 2> 354919 INFO (OverseerCollectionConfigSetProcessor-96706052996988935-127.0.0.1:54784_solr-n_0000000000) [n:127.0.0.1:54784_solr ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper [junit4] 2> 354931 INFO (recoveryExecutor-166-thread-1-processing-n:127.0.0.1:39449_solr x:people_shard2_replica2 s:shard2 c:people r:core_node1) [n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.c.RecoveryStrategy Sending prep recovery command to [http://127.0.0.1:54784/solr]; [WaitForState: action=PREPRECOVERY&core=people_shard2_replica1&nodeName=127.0.0.1:39449_solr&coreNodeName=core_node1&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true] [junit4] 2> 354959 INFO (qtp1231523479-1177) [n:127.0.0.1:54784_solr ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node1, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true [junit4] 2> 354960 INFO (qtp1231523479-1177) [n:127.0.0.1:54784_solr ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see people_shard2_replica1 (shard2 of people) have state: recovering [junit4] 2> 354960 INFO (qtp1231523479-1177) [n:127.0.0.1:54784_solr ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=people, shard=shard2, thisCore=people_shard2_replica1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:39449_solr, coreNodeName=core_node1, onlyIfActiveCheckResult=false, nodeProps: core_node1:{"core":"people_shard2_replica2","base_url":"http://127.0.0.1:39449/solr","node_name":"127.0.0.1:39449_solr","state":"down"} [junit4] 2> 355023 INFO (zkCallback-177-thread-2-processing-n:127.0.0.1:39449_solr) [n:127.0.0.1:39449_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5]) [junit4] 2> 355023 INFO (zkCallback-171-thread-1-processing-n:127.0.0.1:38475_solr) [n:127.0.0.1:38475_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5]) [junit4] 2> 355024 INFO (zkCallback-180-thread-1-processing-n:127.0.0.1:58449_solr) [n:127.0.0.1:58449_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5]) [junit4] 2> 355025 INFO (zkCallback-161-thread-2-processing-n:127.0.0.1:54784_solr) [n:127.0.0.1:54784_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5]) [junit4] 2> 355822 INFO (qtp1663009772-1161) [n:127.0.0.1:38475_solr ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=people, shard=shard1, thisCore=people_shard1_replica2, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:58449_solr, coreNodeName=core_node3, onlyIfActiveCheckResult=false, nodeProps: core_node3:{"core":"people_shard1_replica1","base_url":"http://127.0.0.1:58449/solr","node_name":"127.0.0.1:58449_solr","state":"recovering"} [junit4] 2> 355822 INFO (qtp1663009772-1161) [n:127.0.0.1:38475_solr ] o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds. [junit4] 2> 355822 INFO (qtp1663009772-1161) [n:127.0.0.1:38475_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:58449_solr&onlyIfLeaderActive=true&core=people_shard1_replica2&coreNodeName=core_node3&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=1016 [junit4] 2> 355966 INFO (qtp1231523479-1177) [n:127.0.0.1:54784_solr ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=people, shard=shard2, thisCore=people_shard2_replica1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:39449_solr, coreNodeName=core_node1, onlyIfActiveCheckResult=false, nodeProps: core_node1:{"core":"people_shard2_replica2","base_url":"http://127.0.0.1:39449/solr","node_name":"127.0.0.1:39449_solr","state":"recovering"} [junit4] 2> 355966 INFO (qtp1231523479-1177) [n:127.0.0.1:54784_solr ] o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds. [junit4] 2> 355966 INFO (qtp1231523479-1177) [n:127.0.0.1:54784_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:39449_solr&onlyIfLeaderActive=true&core=people_shard2_replica1&coreNodeName=core_node1&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=1006 [junit4] 2> 362834 INFO (recoveryExecutor-164-thread-1-processing-n:127.0.0.1:58449_solr x:people_shard1_replica1 s:shard1 c:people r:core_node3) [n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.c.RecoveryStrategy Attempting to PeerSync from [http://127.0.0.1:38475/solr/people_shard1_replica2/] - recoveringAfterStartup=[true] [junit4] 2> 362834 INFO (recoveryExecutor-164-thread-1-processing-n:127.0.0.1:58449_solr x:people_shard1_replica1 s:shard1 c:people r:core_node3) [n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.u.PeerSync PeerSync: core=people_shard1_replica1 url=http://127.0.0.1:58449/solr START replicas=[http://127.0.0.1:38475/solr/people_shard1_replica2/] nUpdates=100 [junit4] 2> 362857 INFO (qtp1663009772-1165) [n:127.0.0.1:38475_solr c:people s:shard1 r:core_node2 x:people_shard1_replica2] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0} [junit4] 2> 362858 INFO (qtp1663009772-1165) [n:127.0.0.1:38475_solr c:people s:shard1 r:core_node2 x:people_shard1_replica2] o.a.s.c.S.Request [people_shard1_replica2] webapp=/solr path=/get params={distrib=false&qt=/get&getFingerprint=9223372036854775807&wt=javabin&version=2} status=0 QTime=2 [junit4] 2> 362863 INFO (recoveryExecutor-164-thread-1-processing-n:127.0.0.1:58449_solr x:people_shard1_replica1 s:shard1 c:people r:core_node3) [n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0} [junit4] 2> 362863 INFO (recoveryExecutor-164-thread-1-processing-n:127.0.0.1:58449_solr x:people_shard1_replica1 s:shard1 c:people r:core_node3) [n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.u.PeerSync We are already in sync. No need to do a PeerSync [junit4] 2> 362863 INFO (recoveryExecutor-164-thread-1-processing-n:127.0.0.1:58449_solr x:people_shard1_replica1 s:shard1 c:people r:core_node3) [n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false} [junit4] 2> 362863 INFO (recoveryExecutor-164-thread-1-processing-n:127.0.0.1:58449_solr x:people_shard1_replica1 s:shard1 c:people r:core_node3) [n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit. [junit4] 2> 362874 INFO (recoveryExecutor-164-thread-1-processing-n:127.0.0.1:58449_solr x:people_shard1_replica1 s:shard1 c:people r:core_node3) [n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.u.DirectUpdateHandler2 end_commit_flush [junit4] 2> 362881 INFO (recoveryExecutor-164-thread-1-processing-n:127.0.0.1:58449_solr x:people_shard1_replica1 s:shard1 c:people r:core_node3) [n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.c.RecoveryStrategy PeerSync stage of recovery was successful. [junit4] 2> 362882 INFO (recoveryExecutor-164-thread-1-processing-n:127.0.0.1:58449_solr x:people_shard1_replica1 s:shard1 c:people r:core_node3) [n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.c.RecoveryStrategy Replaying updates buffered during PeerSync. [junit4] 2> 362882 INFO (recoveryExecutor-164-thread-1-processing-n:127.0.0.1:58449_solr x:people_shard1_replica1 s:shard1 c:people r:core_node3) [n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.c.RecoveryStrategy No replay needed. [junit4] 2> 362882 INFO (recoveryExecutor-164-thread-1-processing-n:127.0.0.1:58449_solr x:people_shard1_replica1 s:shard1 c:people r:core_node3) [n:127.0.0.1:58449_solr c:people s:shard1 r:core_node3 x:people_shard1_replica1] o.a.s.c.RecoveryStrategy Registering as Active after recovery. [junit4] 2> 362886 INFO (zkCallback-177-thread-2-processing-n:127.0.0.1:39449_solr) [n:127.0.0.1:39449_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5]) [junit4] 2> 362886 INFO (zkCallback-171-thread-1-processing-n:127.0.0.1:38475_solr) [n:127.0.0.1:38475_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5]) [junit4] 2> 362886 INFO (zkCallback-180-thread-1-processing-n:127.0.0.1:58449_solr) [n:127.0.0.1:58449_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5]) [junit4] 2> 362888 INFO (zkCallback-161-thread-1-processing-n:127.0.0.1:54784_solr) [n:127.0.0.1:54784_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5]) [junit4] 2> 362973 INFO (recoveryExecutor-166-thread-1-processing-n:127.0.0.1:39449_solr x:people_shard2_replica2 s:shard2 c:people r:core_node1) [n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.c.RecoveryStrategy Attempting to PeerSync from [http://127.0.0.1:54784/solr/people_shard2_replica1/] - recoveringAfterStartup=[true] [junit4] 2> 362973 INFO (recoveryExecutor-166-thread-1-processing-n:127.0.0.1:39449_solr x:people_shard2_replica2 s:shard2 c:people r:core_node1) [n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.u.PeerSync PeerSync: core=people_shard2_replica2 url=http://127.0.0.1:39449/solr START replicas=[http://127.0.0.1:54784/solr/people_shard2_replica1/] nUpdates=100 [junit4] 2> 362983 INFO (qtp1231523479-1172) [n:127.0.0.1:54784_solr c:people s:shard2 r:core_node4 x:people_shard2_replica1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0} [junit4] 2> 362983 INFO (qtp1231523479-1172) [n:127.0.0.1:54784_solr c:people s:shard2 r:core_node4 x:people_shard2_replica1] o.a.s.c.S.Request [people_shard2_replica1] webapp=/solr path=/get params={distrib=false&qt=/get&getFingerprint=9223372036854775807&wt=javabin&version=2} status=0 QTime=3 [junit4] 2> 362991 INFO (recoveryExecutor-166-thread-1-processing-n:127.0.0.1:39449_solr x:people_shard2_replica2 s:shard2 c:people r:core_node1) [n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0} [junit4] 2> 362991 INFO (recoveryExecutor-166-thread-1-processing-n:127.0.0.1:39449_solr x:people_shard2_replica2 s:shard2 c:people r:core_node1) [n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.u.PeerSync We are already in sync. No need to do a PeerSync [junit4] 2> 362991 INFO (recoveryExecutor-166-thread-1-processing-n:127.0.0.1:39449_solr x:people_shard2_replica2 s:shard2 c:people r:core_node1) [n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false} [junit4] 2> 362991 INFO (recoveryExecutor-166-thread-1-processing-n:127.0.0.1:39449_solr x:people_shard2_replica2 s:shard2 c:people r:core_node1) [n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit. [junit4] 2> 362997 INFO (recoveryExecutor-166-thread-1-processing-n:127.0.0.1:39449_solr x:people_shard2_replica2 s:shard2 c:people r:core_node1) [n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.u.DirectUpdateHandler2 end_commit_flush [junit4] 2> 363001 INFO (recoveryExecutor-166-thread-1-processing-n:127.0.0.1:39449_solr x:people_shard2_replica2 s:shard2 c:people r:core_node1) [n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.c.RecoveryStrategy PeerSync stage of recovery was successful. [junit4] 2> 363001 INFO (recoveryExecutor-166-thread-1-processing-n:127.0.0.1:39449_solr x:people_shard2_replica2 s:shard2 c:people r:core_node1) [n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.c.RecoveryStrategy Replaying updates buffered during PeerSync. [junit4] 2> 363001 INFO (recoveryExecutor-166-thread-1-processing-n:127.0.0.1:39449_solr x:people_shard2_replica2 s:shard2 c:people r:core_node1) [n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.c.RecoveryStrategy No replay needed. [junit4] 2> 363002 INFO (recoveryExecutor-166-thread-1-processing-n:127.0.0.1:39449_solr x:people_shard2_replica2 s:shard2 c:people r:core_node1) [n:127.0.0.1:39449_solr c:people s:shard2 r:core_node1 x:people_shard2_replica2] o.a.s.c.RecoveryStrategy Registering as Active after recovery. [junit4] 2> 363108 INFO (zkCallback-177-thread-2-processing-n:127.0.0.1:39449_solr) [n:127.0.0.1:39449_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5]) [junit4] 2> 363108 INFO (zkCallback-171-thread-1-processing-n:127.0.0.1:38475_solr) [n:127.0.0.1:38475_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5]) [junit4] 2> 363108 INFO (zkCallback-180-thread-1-processing-n:127.0.0.1:58449_solr) [n:127.0.0.1:58449_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5]) [junit4] 2> 363108 INFO (zkCallback-161-thread-1-processing-n:127.0.0.1:54784_solr) [n:127.0.0.1:54784_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5]) [junit4] 2> 363937 INFO (qtp557959012-1153) [n:127.0.0.1:39449_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=2&property.schema=schema-docValuesJoin.xml&property.config=solrconfig-doctransformers.xml&collection.configName=solrCloudCollectionConfig&name=people&action=CREATE&numShards=2&wt=javabin&version=2} status=0 QTime=11864 [junit4] 2> 363942 INFO (qtp557959012-1154) [n:127.0.0.1:39449_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&property.schema=schema-docValuesJoin.xml&property.config=solrconfig-doctransformers.xml&collection.configName=solrCloudCollectionConfig&name=departments&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true [junit4] 2> 363946 INFO (OverseerThreadFactory-448-thread-2-processing-n:127.0.0.1:54784_solr) [n:127.0.0.1:54784_solr ] o.a.s.c.CreateCollectionCmd Create collection departments [junit4] 2> 364382 INFO (qtp1663009772-1164) [n:127.0.0.1:38475_solr ] o.a.s.h.a.CoreAdminOperation core create command property.schema=schema-docValuesJoin.xml&property.config=solrconfig-doctransformers.xml&qt=/admin/cores&collection.configName=solrCloudCollectionConfig&name=departments_shard2_replica1&action=CREATE&numShards=2&collection=departments&shard=shard2&wt=javabin&version=2 [junit4] 2> 364383 INFO (qtp557959012-1156) [n:127.0.0.1:39449_solr ] o.a.s.h.a.CoreAdminOperation core create command property.schema=schema-docValuesJoin.xml&property.config=solrconfig-doctransformers.xml&qt=/admin/cores&collection.configName=solrCloudCollectionConfig&name=departments_shard2_replica2&action=CREATE&numShards=2&collection=departments&shard=shard2&wt=javabin&version=2 [junit4] 2> 364385 INFO (qtp1231523479-1175) [n:127.0.0.1:54784_solr ] o.a.s.h.a.CoreAdminOperation core create command property.schema=schema-docValuesJoin.xml&property.config=solrconfig-doctransformers.xml&qt=/admin/cores&collection.configName=solrCloudCollectionConfig&name=departments_shard1_replica2&action=CREATE&numShards=2&collection=departments&shard=shard1&wt=javabin&version=2 [junit4] 2> 364399 INFO (qtp444453979-1197) [n:127.0.0.1:34130_solr ] o.a.s.h.a.CoreAdminOperation core create command property.schema=schema-docValuesJoin.xml&property.config=solrconfig-doctransformers.xml&qt=/admin/cores&collection.configName=solrCloudCollectionConfig&name=departments_shard1_replica1&action=CREATE&numShards=2&collection=departments&shard=shard1&wt=javabin&version=2 [junit4] 2> 364531 INFO (zkCallback-161-thread-2-processing-n:127.0.0.1:54784_solr) [n:127.0.0.1:54784_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/departments/state.json] for collection [departments] has occurred - updating... (live nodes size: [5]) [junit4] 2> 364532 INFO (zkCallback-178-thread-1-processing-n:127.0.0.1:34130_solr) [n:127.0.0.1:34130_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/departments/state.json] for collection [departments] has occurred - updating... (live nodes size: [5]) [junit4] 2> 364532 INFO (zkCallback-177-thread-2-processing-n:127.0.0.1:39449_solr) [n:127.0.0.1:39449_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/departments/state.json] for collection [departments] has occurred - updating... (live nodes size: [5]) [junit4] 2> 364532 INFO (zkCallback-171-thread-1-processing-n:127.0.0.1:38475_solr) [n:127.0.0.1:38475_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/departments/state.json] for collection [departments] has occurred - updating... (live nodes size: [5]) [junit4] 2> 365449 INFO (qtp1231523479-1175) [n:127.0.0.1:54784_solr c:departments s:shard1 x:departments_shard1_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.3.0 [junit4] 2> 365479 INFO (qtp557959012-1156) [n:127.0.0.1:39449_solr c:departments s:shard2 x:departments_shard2_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.3.0 [junit4] 2> 365519 INFO (qtp444453979-1197) [n:127.0.0.1:34130_solr c:departments s:shard1 x:departments_shard1_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.3.0 [junit4] 2> 365524 INFO (qtp1663009772-1164) [n:127.0.0.1:38475_solr c:departments s:shard2 x:departments_shard2_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.3.0 [junit4] 2> 365535 INFO (qtp1231523479-1175) [n:127.0.0.1:54784_solr c:departments s:shard1 x:departments_shard1_replica2] o.a.s.s.IndexSchema [departments_shard1_replica2] Schema name=doc-values-for-Join [junit4] 2> 365678 WARN (qtp1231523479-1175) [n:127.0.0.1:54784_solr c:departments s:shard1 x:departments_shard1_replica2] o.a.s.s.IndexSchema [departments_shard1_replica2] default search field in schema is id. WARNING: Deprecated, please use 'df' on request instead. [junit4] 2> 365679 INFO (qtp1231523479-1175) [n:127.0.0.1:54784_solr c:departments s:shard1 x:departments_shard1_replica2] o.a.s.s.IndexSchema Loaded schema doc-values-for-Join/1.6 with uniqueid field id [junit4] 2> 365679 INFO (qtp557959012-1156) [n:127.0.0.1:39449_solr c:departments s:shard2 x:departments_shard2_replica2] o.a.s.s.IndexSchema [departments_shard2_replica2] Schema name=doc-values-for-Join [junit4] 2> 365687 WARN (qtp557959012-1156) [n:127.0.0.1:39449_solr c:departments s:shard2 x:departments_shard2_replica2] o.a.s.s.IndexSchema [departments_shard2_replica2] default search field in schema is id. WARNING: Deprecated, please use 'df' on request instead. [junit4] 2> 365688 INFO (qtp557959012-1156) [n:127.0.0.1:39449_solr c:departments s:shard2 x:departments_shard2_replica2] o.a.s.s.IndexSchema Loaded schema doc-values-for-Join/1.6 with uniqueid field id [junit4] 2> 365695 INFO (qtp1663009772-1164) [n:127.0.0.1:38475_solr c:departments s:shard2 x:departments_shard2_replica1] o.a.s.s.IndexSchema [departments_shard2_replica1] Schema name=doc-values-for-Join [junit4] 2> 365707 INFO (qtp444453979-1197) [n:127.0.0.1:34130_solr c:departments s:shard1 x:departments_shard1_replica1] o.a.s.s.IndexSchema [departments_shard1_replica1] Schema name=doc-values-for-Join [junit4] 2> 365709 INFO (qtp1231523479-1175) [n:127.0.0.1:54784_solr c:departments s:shard1 x:departments_shard1_replica2] o.a.s.c.CoreContainer Creating SolrCore 'departments_shard1_replica2' using configuration from collection departments [junit4] 2> 365709 INFO (qtp1231523479-1175) [n:127.0.0.1:54784_solr c:departments s:shard1 r:core_node2 x:departments_shard1_replica2] o.a.s.c.SolrCore [[departments_shard1_replica2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node2/departments_shard1_replica2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node2/./departments_shard1_replica2/data/] [junit4] 2> 365722 INFO (qtp557959012-1156) [n:127.0.0.1:39449_solr c:departments s:shard2 x:departments_shard2_replica2] o.a.s.c.CoreContainer Creating SolrCore 'departments_shard2_replica2' using configuration from collection departments [junit4] 2> 365722 INFO (qtp557959012-1156) [n:127.0.0.1:39449_solr c:departments s:shard2 r:core_node1 x:departments_shard2_replica2] o.a.s.c.SolrCore [[departments_shard2_replica2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node3/departments_shard2_replica2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node3/./departments_shard2_replica2/data/] [junit4] 2> 365729 WARN (qtp1663009772-1164) [n:127.0.0.1:38475_solr c:departments s:shard2 x:departments_shard2_replica1] o.a.s.s.IndexSchema [departments_shard2_replica1] default search field in schema is id. WARNING: Deprecated, please use 'df' on request instead. [junit4] 2> 365729 INFO (qtp1663009772-1164) [n:127.0.0.1:38475_solr c:departments s:shard2 x:departments_shard2_replica1] o.a.s.s.IndexSchema Loaded schema doc-values-for-Join/1.6 with uniqueid field id [junit4] 2> 365732 WARN (qtp444453979-1197) [n:127.0.0.1:34130_solr c:departments s:shard1 x:departments_shard1_replica1] o.a.s.s.IndexSchema [departments_shard1_replica1] default search field in schema is id. WARNING: Deprecated, please use 'df' on request instead. [junit4] 2> 365733 INFO (qtp444453979-1197) [n:127.0.0.1:34130_solr c:departments s:shard1 x:departments_shard1_replica1] o.a.s.s.IndexSchema Loaded schema doc-values-for-Join/1.6 with uniqueid field id [junit4] 2> 365804 INFO (qtp1663009772-1164) [n:127.0.0.1:38475_solr c:departments s:shard2 x:departments_shard2_replica1] o.a.s.c.CoreContainer Creating SolrCore 'departments_shard2_replica1' using configuration from collection departments [junit4] 2> 365805 INFO (qtp1663009772-1164) [n:127.0.0.1:38475_solr c:departments s:shard2 r:core_node3 x:departments_shard2_replica1] o.a.s.c.SolrCore [[departments_shard2_replica1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node1/departments_shard2_replica1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node1/./departments_shard2_replica1/data/] [junit4] 2> 365806 INFO (qtp444453979-1197) [n:127.0.0.1:34130_solr c:departments s:shard1 x:departments_shard1_replica1] o.a.s.c.CoreContainer Creating SolrCore 'departments_shard1_replica1' using configuration from collection departments [junit4] 2> 365806 INFO (qtp444453979-1197) [n:127.0.0.1:34130_solr c:departments s:shard1 r:core_node4 x:departments_shard1_replica1] o.a.s.c.SolrCore [[departments_shard1_replica1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node5/departments_shard1_replica1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.response.transform.TestSubQueryTransformerDistrib_B6C73D6CCF57C41A-001/tempDir-001/node5/./departments_shard1_replica1/data/] [junit4] 2> 365861 INFO (qtp1231523479-1175) [n:127.0.0.1:54784_solr c:departments s:shard1 r:core_node2 x:departments_shard1_replica2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog [junit4] 2> 365861 INFO (qtp1231523479-1175) [n:127.0.0.1:54784_solr c:departments s:shard1 r:core_node2 x:departments_shard1_replica2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 [junit4] 2> 365862 INFO (qtp1231523479-1175) [n:127.0.0.1:54784_solr c:departments s:shard1 r:core_node2 x:departments_shard1_replica2] o.a.s.u.CommitTracker Hard AutoCommit: disabled [junit4] 2> 365862 INFO (qtp1231523479-1175) [n:127.0.0.1:54784_solr c:departments s:shard1 r:core_node2 x:departments_shard1_replica2] o.a.s.u.CommitTracker Soft AutoCommit: disabled [junit4] 2> 365870 INFO (qtp557959012-1156) [n:127.0.0.1:39449_solr c:departments s:shard2 r:core_node1 x:departments_shard2_replica2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog [junit4] 2> 365870 INFO (qtp557959012-1156) [n:127.0.0.1:39449_solr c:departments s:shard2 r:core_node1 x:departments_shard2_replica2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 [junit4] 2> 365871 INFO (qtp557959012-1156) [n:127.0.0.1:39449_solr c:departments s:shard2 r:core_node1 x:departments_shard2_replica2] o.a.s.u.CommitTracker Hard AutoCommit: disabled [junit4] 2> 365871 INFO (qtp557959012-1156) [n:127.0.0.1:39449_solr c:departments s:shard2 r:core_node1 x:departments_shard2_replica2] o.a.s.u.CommitTracker Soft AutoCommit: disabled [junit4] 2> 365871 INFO (qtp1231523479-1175) [n:127.0.0.1:54784_solr c:departments s:shard1 r:core_node2 x:departments_shard1_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@48e4f4ff[departments_shard1_replica2] main] [junit4] 2> 365872 INFO (qtp557959012-1156) [n:127.0.0.1:39449_solr c:departments s:shard2 r:core_node1 x:departments_shard2_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@4a060658[departments_shard2_replica2] main] [junit4] 2> 365874 INFO (qtp557959012-1156) [n:127.0.0.1:39449_solr c:departments s:shard2 r:core_node1 x:departments_shard2_replica2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/solrCloudCollectionConfig [junit4] 2> 365874 INFO (qtp557959012-1156) [n:127.0.0.1:39449_solr c:departments s:shard2 r:core_node1 x:departments_shard2_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/solrCloudCollectionConfig [junit4] 2> 365874 INFO (qtp557959012-1156) [n:127.0.0.1:39449_solr c:departments s:shard2 r:core_node1 x:departments_shard2_replica2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 [junit4] 2> 365875 INFO (qtp1231523479-1175) [n:127.0.0.1:54784_solr c:departments s:shard1 r:core_node2 x:departments_shard1_replica2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/solrCloudCollectionConfig [junit4] 2> 365875 INFO (qtp1231523479-1175) [n:127.0.0.1:54784_solr c:departments s:shard1 r:core_node2 x:departments_shard1_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/solrCloudCollectionConfig [junit4] 2> 365875 INFO (qtp1231523479-1175) [n:127.0.0.1:54784_solr c:departments s:shard1 r:core_node2 x:departments_shard1_replica2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 [junit4] 2> 365950 INFO (qtp1231523479-1175) [n:127.0.0.1:54784_solr c:departments s:shard1 r:core_node2 x:departments_shard1_replica2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1547296866862891008 [junit4] 2> 365952 INFO (qtp557959012-1156) [n:127.0.0.1:39449_solr c:departments s:shard2 r:core_node1 x:departments_shard2_replica2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1547296866864988160 [junit4] 2> 365962 INFO (searcherExecutor-477-thread-1-processing-n:127.0.0.1:54784_solr x:departments_shard1_replica2 s:shard1 c:departments r:core_node2) [n:127.0.0.1:54784_solr c:departments s:shard1 r:core_node2 x:departments_shard1_replica2] o.a.s.c.SolrCore [departments_shard1_replica2] Registered new searcher Searcher@48e4f4ff[departments_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())} [junit4] 2> 365976 INFO (qtp557959012-1156) [n:127.0.0.1:39449_solr c:departments s:shard2 r:core_node1 x:departments_shard2_replica2] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoutin=9999ms [junit4] 2> 365978 INFO (qtp1231523479-1175) [n:127.0.0.1:54784_solr c:departments s:shard1 r:core_node2 x:departments_shard1_replica2] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=9999ms [junit4] 2> 365979 INFO (searcherExecutor-478-thread-1-processing-n:127.0.0.1:39449_solr x:departments_shard2_replica2 s:shard2 c:departments r:core_node1) [n:127.0.0.1:39449_solr c:departments s:shard2 r:core_node1 x:departments_shard2_replica2] o.a.s.c.SolrCore [departments_shard2_replica2] Registered new searcher Searcher@4a060658[departments_shard2_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())} [junit4] 2> 366002 INFO (qtp444453979-1197) [n:127.0.0.1:34130_solr c:departments s:shard1 r:core_node4 x:departments_shard1_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog [junit4] 2> 366002 INFO (qtp444453979-1197) [n:127.0.0.1:34130_solr c:departments s:shard1 r:core_node4 x:departments_shard1_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 [junit4] 2> 366003 INFO (qtp444453979-1197) [n:127.0.0.1:34130_solr c:departm [...truncated too long message...] ate.json [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:322) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:319) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:319) [junit4] 2> at org.apache.solr.cloud.overseer.ZkStateWriter.writePendingUpdates(ZkStateWriter.java:228) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:225) [junit4] 2> at java.lang.Thread.run(Thread.java:745) [junit4] 2> 377638 INFO (OverseerStateUpdate-96706052996988935-127.0.0.1:54784_solr-n_0000000000) [n:127.0.0.1:54784_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:54784_solr [junit4] 2> 377641 INFO (zkCallback-171-thread-1-processing-n:127.0.0.1:38475_solr) [n:127.0.0.1:38475_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (2) [junit4] 2> 377669 INFO (zkCallback-171-thread-3-processing-n:127.0.0.1:38475_solr) [n:127.0.0.1:38475_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1) [junit4] 2> 377760 INFO (OverseerStateUpdate-96706052996988937-127.0.0.1:38475_solr-n_0000000001) [n:127.0.0.1:38475_solr ] o.a.s.c.Overseer According to ZK I (id=96706052996988937-127.0.0.1:38475_solr-n_0000000001) am no longer a leader. [junit4] 2> 377760 INFO (OverseerStateUpdate-96706052996988937-127.0.0.1:38475_solr-n_0000000001) [n:127.0.0.1:38475_solr ] o.a.s.c.Overseer According to ZK I (id=96706052996988937-127.0.0.1:38475_solr-n_0000000001) am no longer a leader. [junit4] 2> 377760 INFO (OverseerStateUpdate-96706052996988937-127.0.0.1:38475_solr-n_0000000001) [n:127.0.0.1:38475_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:38475_solr [junit4] 2> 377762 INFO (OverseerCollectionConfigSetProcessor-96706052996988937-127.0.0.1:38475_solr-n_0000000001) [n:127.0.0.1:38475_solr ] o.a.s.c.OverseerTaskProcessor According to ZK I (id=96706052996988937-127.0.0.1:38475_solr-n_0000000001) am no longer a leader. [junit4] 2> 377762 ERROR (OverseerCollectionConfigSetProcessor-96706052996988937-127.0.0.1:38475_solr-n_0000000001) [n:127.0.0.1:38475_solr ] o.a.s.c.OverseerTaskProcessor :org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/collection-queue-work [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:339) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:336) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:336) [junit4] 2> at org.apache.solr.cloud.DistributedQueue.fetchZkChildren(DistributedQueue.java:299) [junit4] 2> at org.apache.solr.cloud.OverseerTaskQueue.getTailId(OverseerTaskQueue.java:264) [junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:162) [junit4] 2> at java.lang.Thread.run(Thread.java:745) [junit4] 2> [junit4] 2> 377763 INFO (OverseerCollectionConfigSetProcessor-96706052996988937-127.0.0.1:38475_solr-n_0000000001) [n:127.0.0.1:38475_solr ] o.a.s.c.OverseerTaskProcessor According to ZK I (id=96706052996988937-127.0.0.1:38475_solr-n_0000000001) am no longer a leader. [junit4] 2> 377766 INFO (jetty-launcher-151-thread-3) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@4cd97e8e{/solr,null,UNAVAILABLE} [junit4] 2> 377771 WARN (zkCallback-171-thread-3-processing-n:127.0.0.1:38475_solr) [n:127.0.0.1:38475_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes] [junit4] 2> 377790 INFO (jetty-launcher-151-thread-4) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@2e50d386{/solr,null,UNAVAILABLE} [junit4] 2> 378992 WARN (zkCallback-180-thread-1-processing-n:127.0.0.1:58449_solr) [n:127.0.0.1:58449_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes] [junit4] 2> 378993 INFO (jetty-launcher-151-thread-1) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@75f1e9bd{/solr,null,UNAVAILABLE} [junit4] 2> 379095 INFO (jetty-launcher-151-thread-2) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@6a6770cc{/solr,null,UNAVAILABLE} [junit4] 2> 379098 INFO (SUITE-TestSubQueryTransformerDistrib-seed#[B6C73D6CCF57C41A]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:51643 51643 [junit4] 2> 379206 INFO (Thread-392) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:51643 51643 [junit4] 2> 379207 WARN (Thread-392) [ ] o.a.s.c.ZkTestServer Watch limit violations: [junit4] 2> Maximum concurrent create/delete watches above limit: [junit4] 2> [junit4] 2> 6 /solr/aliases.json [junit4] 2> 6 /solr/clusterprops.json [junit4] 2> 5 /solr/security.json [junit4] 2> 5 /solr/configs/solrCloudCollectionConfig [junit4] 2> [junit4] 2> Maximum concurrent data watches above limit: [junit4] 2> [junit4] 2> 6 /solr/clusterstate.json [junit4] 2> 4 /solr/collections/departments/state.json [junit4] 2> 4 /solr/collections/people/state.json [junit4] 2> [junit4] 2> Maximum concurrent children watches above limit: [junit4] 2> [junit4] 2> 21 /solr/overseer/queue [junit4] 2> 21 /solr/overseer/collection-queue-work [junit4] 2> 8 /solr/overseer/queue-work [junit4] 2> 6 /solr/live_nodes [junit4] 2> 6 /solr/collections [junit4] 2> [junit4] 2> 379207 INFO (SUITE-TestSubQueryTransformerDistrib-seed#[B6C73D6CCF57C41A]-worker) [ ] o.a.s.SolrTestCaseJ4 ###deleteCore [junit4] 2> Oct 04, 2016 9:41:35 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks [junit4] 2> WARNING: Will linger awaiting termination of 1 leaked thread(s). [junit4] 2> Oct 04, 2016 9:41:55 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks [junit4] 2> SEVERE: 1 thread leaked from SUITE scope at org.apache.solr.response.transform.TestSubQueryTransformerDistrib: [junit4] 2> 1) Thread[id=1369, name=OverseerHdfsCoreFailoverThread-96706052996988937-127.0.0.1:38475_solr-n_0000000001, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.] [junit4] 2> at java.lang.Thread.sleep(Native Method) [junit4] 2> at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137) [junit4] 2> at java.lang.Thread.run(Thread.java:745) [junit4] 2> Oct 04, 2016 9:41:55 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll [junit4] 2> INFO: Starting to interrupt leaked threads: [junit4] 2> 1) Thread[id=1369, name=OverseerHdfsCoreFailoverThread-96706052996988937-127.0.0.1:38475_solr-n_0000000001, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.] [junit4] 2> Oct 04, 2016 9:41:55 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll [junit4] 2> INFO: All leaked threads terminated. [junit4] 2> NOTE: test params are: codec=CheapBastard, sim=ClassicSimilarity, locale=sv, timezone=Pacific/Norfolk [junit4] 2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_102 (64-bit)/cpus=4,threads=1,free=69174440,total=368050176 [junit4] 2> NOTE: All tests run in this JVM: [TestIndexingPerformance, CdcrUpdateLogTest, CurrencyFieldXmlFileTest, TestLeaderElectionZkExpiry, DeleteInactiveReplicaTest, TestPhraseSuggestions, TestExtendedDismaxParser, TestSolrQueryParser, TestRestoreCore, TestHdfsBackupRestoreCore, TestAnalyzeInfixSuggestions, TestJettySolrRunner, TestRandomFlRTGCloud, TestPerFieldSimilarityWithDefaultOverride, TestReload, QueryParsingTest, UpdateParamsTest, TestAuthorizationFramework, TestSubQueryTransformerDistrib] [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestSubQueryTransformerDistrib -Dtests.seed=B6C73D6CCF57C41A -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=sv -Dtests.timezone=Pacific/Norfolk -Dtests.asserts=true -Dtests.file.encoding=UTF-8 [junit4] ERROR 0.00s J2 | TestSubQueryTransformerDistrib (suite) <<< [junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.response.transform.TestSubQueryTransformerDistrib: [junit4] > 1) Thread[id=1369, name=OverseerHdfsCoreFailoverThread-96706052996988937-127.0.0.1:38475_solr-n_0000000001, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.] [junit4] > at java.lang.Thread.sleep(Native Method) [junit4] > at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137) [junit4] > at java.lang.Thread.run(Thread.java:745) [junit4] > at __randomizedtesting.SeedInfo.seed([B6C73D6CCF57C41A]:0) [junit4] Completed [50/637 (1!)] on J2 in 51.56s, 1 test, 1 error <<< FAILURES! [...truncated 62190 lines...]
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org