Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/816/

3 tests failed.
FAILED:  
org.apache.solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR

Error Message:
Captured an uncaught exception in thread: Thread[id=13321, 
name=coreZkRegister-3100-thread-2, state=RUNNABLE, 
group=TGRP-LeaderInitiatedRecoveryOnShardRestartTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=13321, name=coreZkRegister-3100-thread-2, 
state=RUNNABLE, group=TGRP-LeaderInitiatedRecoveryOnShardRestartTest]
Caused by: java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([31ADC7DE93344FFB]:0)
        at 
org.apache.solr.cloud.ZkController.updateLeaderInitiatedRecoveryState(ZkController.java:2126)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:433)
        at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:197)
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)
        at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:346)
        at 
org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1113)
        at org.apache.solr.cloud.ZkController.register(ZkController.java:926)
        at org.apache.solr.cloud.ZkController.register(ZkController.java:881)
        at org.apache.solr.core.ZkContainer$2.run(ZkContainer.java:183)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  
org.apache.solr.cloud.OverseerTest.testExternalClusterStateChangeBehavior

Error Message:
Illegal state, was: down expected:active clusterState:live 
nodes:[]collections:{c1=DocCollection(c1)={   "shards":{"shard1":{       
"parent":null,       "range":null,       "state":"active",       
"replicas":{"core_node1":{           "base_url":"http://127.0.0.1/solr";,        
   "node_name":"node1",           "core":"core1",           "roles":"",         
  "state":"down"}}}},   "router":{"name":"implicit"}}, 
test=LazyCollectionRef(test)}

Stack Trace:
java.lang.AssertionError: Illegal state, was: down expected:active 
clusterState:live nodes:[]collections:{c1=DocCollection(c1)={
  "shards":{"shard1":{
      "parent":null,
      "range":null,
      "state":"active",
      "replicas":{"core_node1":{
          "base_url":"http://127.0.0.1/solr";,
          "node_name":"node1",
          "core":"core1",
          "roles":"",
          "state":"down"}}}},
  "router":{"name":"implicit"}}, test=LazyCollectionRef(test)}
        at 
__randomizedtesting.SeedInfo.seed([31ADC7DE93344FFB:59B3C43271A415B5]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at 
org.apache.solr.cloud.OverseerTest.verifyStatus(OverseerTest.java:601)
        at 
org.apache.solr.cloud.OverseerTest.testExternalClusterStateChangeBehavior(OverseerTest.java:1261)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1665)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:864)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:900)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:914)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:873)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:775)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:809)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:820)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Error from server at http://127.0.0.1:48215: Could not load collection from 
ZK:nodes_used_collection

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at http://127.0.0.1:48215: Could not load collection from 
ZK:nodes_used_collection
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:575)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:302)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:419)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:169)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1665)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:864)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:900)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:914)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:963)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:873)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:775)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:809)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:820)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 10210 lines...]
   [junit4] Suite: 
org.apache.solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest
   [junit4]   2> Creating dataDir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_31ADC7DE93344FFB-001/init-core-data-001
   [junit4]   2> 1212994 INFO  
(SUITE-LeaderInitiatedRecoveryOnShardRestartTest-seed#[31ADC7DE93344FFB]-worker)
 [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system 
property: /
   [junit4]   2> 1212996 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1212997 INFO  (Thread-7921) [    ] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1212997 INFO  (Thread-7921) [    ] o.a.s.c.ZkTestServer 
Starting server
   [junit4]   2> 1213097 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.ZkTestServer start zk server on port:41939
   [junit4]   2> 1213097 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1213098 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1213100 INFO  (zkCallback-1201-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@13475f name:ZooKeeperConnection 
Watcher:127.0.0.1:41939 got event WatchedEvent state:SyncConnected type:None 
path:null path:null type:None
   [junit4]   2> 1213100 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1213101 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1213101 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 1213105 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1213106 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1213106 INFO  (zkCallback-1202-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@29c079bf 
name:ZooKeeperConnection Watcher:127.0.0.1:41939/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1213107 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1213107 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1213107 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 1213109 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 1213110 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 1213111 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 1213112 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
 to /configs/conf1/solrconfig.xml
   [junit4]   2> 1213112 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1213114 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema.xml
 to /configs/conf1/schema.xml
   [junit4]   2> 1213114 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 1213116 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
 to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1213116 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.SolrZkClient makePath: 
/configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1213122 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
 to /configs/conf1/stopwords.txt
   [junit4]   2> 1213122 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1213123 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt
 to /configs/conf1/protwords.txt
   [junit4]   2> 1213124 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1213125 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml
 to /configs/conf1/currency.xml
   [junit4]   2> 1213125 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 1213127 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
 to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1213127 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1213128 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
 to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1213128 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1213130 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
 to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1213130 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.SolrZkClient makePath: 
/configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1213131 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
 to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1213131 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1213133 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
 to /configs/conf1/synonyms.txt
   [junit4]   2> 1213133 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1213431 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_31ADC7DE93344FFB-001/control-001/cores/collection1
   [junit4]   2> 1213432 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.e.j.s.Server jetty-9.2.13.v20150730
   [junit4]   2> 1213434 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@5bc7fc5e{/,null,AVAILABLE}
   [junit4]   2> 1213435 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.e.j.s.ServerConnector Started 
ServerConnector@63723eb{HTTP/1.1}{127.0.0.1:41759}
   [junit4]   2> 1213435 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.e.j.s.Server Started @1216754ms
   [junit4]   2> 1213436 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_31ADC7DE93344FFB-001/tempDir-001/control/data,
 hostContext=/, hostPort=41759, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_31ADC7DE93344FFB-001/control-001/cores}
   [junit4]   2> 1213436 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): 
sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 1213436 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_31ADC7DE93344FFB-001/control-001/'
   [junit4]   2> 1213451 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1213451 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1213452 INFO  (zkCallback-1203-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@6b12f83d 
name:ZooKeeperConnection Watcher:127.0.0.1:41939/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1213452 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1213453 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1213454 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 1213454 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_31ADC7DE93344FFB-001/control-001/solr.xml
   [junit4]   2> 1213458 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.CoresLocator Config-defined core root directory: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_31ADC7DE93344FFB-001/control-001/cores
   [junit4]   2> 1213459 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.CoreContainer New CoreContainer 1769729160
   [junit4]   2> 1213459 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer 
[instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_31ADC7DE93344FFB-001/control-001/]
   [junit4]   2> 1213459 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.CoreContainer loading shared library: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_31ADC7DE93344FFB-001/control-001/lib
   [junit4]   2> 1213459 WARN  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.SolrResourceLoader Can't find (or read) directory to add to 
classloader: lib (resolved as: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_31ADC7DE93344FFB-001/control-001/lib).
   [junit4]   2> 1213465 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 
90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 
20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 
2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : 
false,useRetries : false,
   [junit4]   2> 1213467 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with 
params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1213467 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1213467 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j 
(org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1213467 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41939/solr
   [junit4]   2> 1213467 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1213467 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1213468 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1213469 INFO  (zkCallback-1205-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@47ac98 name:ZooKeeperConnection 
Watcher:127.0.0.1:41939 got event WatchedEvent state:SyncConnected type:None 
path:null path:null type:None
   [junit4]   2> 1213469 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1213469 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1213471 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:41759_    ] o.a.s.c.c.ConnectionManager Waiting for client to 
connect to ZooKeeper
   [junit4]   2> 1213471 INFO  
(zkCallback-1206-thread-1-processing-n:127.0.0.1:41759_) [n:127.0.0.1:41759_    
] o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@3928e11c 
name:ZooKeeperConnection Watcher:127.0.0.1:41939/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1213471 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:41759_    ] o.a.s.c.c.ConnectionManager Client is connected to 
ZooKeeper
   [junit4]   2> 1213472 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:41759_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 1213474 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:41759_    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-queue-work
   [junit4]   2> 1213476 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:41759_    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-map-running
   [junit4]   2> 1213478 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:41759_    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-map-completed
   [junit4]   2> 1213479 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:41759_    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-map-failure
   [junit4]   2> 1213481 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:41759_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 1213482 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:41759_    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 1213483 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:41759_    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 1213484 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:41759_    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 1213485 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:41759_    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:41759_
   [junit4]   2> 1213485 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:41759_    ] o.a.s.c.c.SolrZkClient makePath: 
/live_nodes/127.0.0.1:41759_
   [junit4]   2> 1213487 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:41759_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 1213488 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:41759_    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer_elect/election
   [junit4]   2> 1213489 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:41759_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1213489 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:41759_    ] o.a.s.c.LeaderElector Joined leadership election with 
path: /overseer_elect/election/94654612068892676-127.0.0.1:41759_-n_0000000000
   [junit4]   2> 1213490 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:41759_    ] o.a.s.c.OverseerElectionContext I am going to be the 
leader 127.0.0.1:41759_
   [junit4]   2> 1213490 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:41759_    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer_elect/leader
   [junit4]   2> 1213491 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:41759_    ] o.a.s.c.Overseer Overseer 
(id=94654612068892676-127.0.0.1:41759_-n_0000000000) starting
   [junit4]   2> 1213492 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:41759_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 1213497 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:41759_    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting 
OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 
autoReplicaFailoverWaitAfterExpiration=30000 
autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1213497 INFO  
(OverseerCollectionConfigSetProcessor-94654612068892676-127.0.0.1:41759_-n_0000000000)
 [n:127.0.0.1:41759_    ] o.a.s.c.OverseerTaskProcessor Process current queue 
of overseer operations
   [junit4]   2> 1213497 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:41759_    ] o.a.s.c.c.ZkStateReader Updating cluster state from 
ZooKeeper... 
   [junit4]   2> 1213497 INFO  
(OverseerStateUpdate-94654612068892676-127.0.0.1:41759_-n_0000000000) 
[n:127.0.0.1:41759_    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 1213515 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:41759_    ] o.a.s.c.CoreContainer Security conf doesn't exist. 
Skipping setup for authorization module.
   [junit4]   2> 1213515 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:41759_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1213515 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:41759_    ] o.a.s.c.CoresLocator Looking for core definitions 
underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_31ADC7DE93344FFB-001/control-001/cores
   [junit4]   2> 1213516 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:41759_    ] o.a.s.c.SolrCore Created CoreDescriptor: 
{name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, 
loadOnStartup=true, configSetProperties=configsetprops.json, 
instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_31ADC7DE93344FFB-001/control-001/cores/collection1,
 collection=control_collection, 
absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_31ADC7DE93344FFB-001/control-001/cores/collection1/,
 coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 1213516 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:41759_    ] o.a.s.c.CoresLocator Found core collection1 in 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_31ADC7DE93344FFB-001/control-001/cores/collection1/
   [junit4]   2> 1213516 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:41759_    ] o.a.s.c.CoresLocator Found 1 core definitions
   [junit4]   2> 1213517 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:41759_    ] o.a.s.s.SolrDispatchFilter 
user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2
   [junit4]   2> 1213517 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:41759_    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() 
done
   [junit4]   2> 1213517 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection   x:collection1] o.a.s.c.ZkController 
publishing state=down
   [junit4]   2> 1213517 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection   x:collection1] o.a.s.c.ZkController 
numShards not found on descriptor - reading it from system property
   [junit4]   2> 1213519 INFO  
(OverseerStateUpdate-94654612068892676-127.0.0.1:41759_-n_0000000000) 
[n:127.0.0.1:41759_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message 
= {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:41759";,
   [junit4]   2>   "node_name":"127.0.0.1:41759_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 1213519 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection   x:collection1] o.a.s.c.ZkController 
look for our core node name
   [junit4]   2> 1213520 INFO  
(OverseerStateUpdate-94654612068892676-127.0.0.1:41759_-n_0000000000) 
[n:127.0.0.1:41759_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 
message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:41759";,
   [junit4]   2>   "node_name":"127.0.0.1:41759_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1213520 INFO  
(OverseerStateUpdate-94654612068892676-127.0.0.1:41759_-n_0000000000) 
[n:127.0.0.1:41759_    ] o.a.s.c.o.ClusterStateMutator building a new cName: 
control_collection
   [junit4]   2> 1213520 INFO  
(OverseerStateUpdate-94654612068892676-127.0.0.1:41759_-n_0000000000) 
[n:127.0.0.1:41759_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard 
shard=shard1
   [junit4]   2> 1213522 INFO  
(zkCallback-1206-thread-1-processing-n:127.0.0.1:41759_) [n:127.0.0.1:41759_    
] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent 
state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred 
- updating... (live nodes size: 1)
   [junit4]   2> 1214520 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection   x:collection1] o.a.s.c.ZkController 
waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1214520 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection   x:collection1] o.a.s.c.ZkController 
Check for collection zkNode:control_collection
   [junit4]   2> 1214521 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection   x:collection1] o.a.s.c.ZkController 
Collection zkNode exists
   [junit4]   2> 1214521 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection   x:collection1] 
o.a.s.c.c.ZkStateReader Load collection config 
from:/collections/control_collection
   [junit4]   2> 1214521 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection   x:collection1] 
o.a.s.c.c.ZkStateReader path=/collections/control_collection configName=conf1 
specified config exists in ZooKeeper
   [junit4]   2> 1214522 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection   x:collection1] 
o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_31ADC7DE93344FFB-001/control-001/cores/collection1/'
   [junit4]   2> 1214535 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection   x:collection1] o.a.s.c.Config loaded 
config solrconfig.xml with version 0 
   [junit4]   2> 1214539 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection   x:collection1] o.a.s.c.SolrConfig 
current version of requestparams : -1
   [junit4]   2> 1214543 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection   x:collection1] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1214551 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection   x:collection1] o.a.s.c.Config Loaded 
SolrConfig: solrconfig.xml
   [junit4]   2> 1214552 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection   x:collection1] o.a.s.s.IndexSchema 
Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 1214557 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection   x:collection1] o.a.s.s.IndexSchema 
[collection1] Schema name=test
   [junit4]   2> 1214664 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Initialized with 
rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1214670 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection   x:collection1] o.a.s.s.IndexSchema 
default search field in schema is text
   [junit4]   2> 1214671 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection   x:collection1] o.a.s.s.IndexSchema 
unique key field: id
   [junit4]   2> 1214679 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection   x:collection1] 
o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1214682 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection   x:collection1] 
o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1214684 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from 
open-exchange-rates.json
   [junit4]   2> 1214684 WARN  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 1214685 WARN  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 1214685 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from 
open-exchange-rates.json
   [junit4]   2> 1214685 WARN  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 1214685 WARN  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 1214687 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection   x:collection1] 
o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default 
properties: Can't find resource 'configsetprops.json' in classpath or 
'/configs/conf1', 
cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2
   [junit4]   2> 1214687 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection   x:collection1] o.a.s.c.CoreContainer 
Creating SolrCore 'collection1' using configuration from collection 
control_collection
   [junit4]   2> 1214687 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1214687 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_31ADC7DE93344FFB-001/control-001/cores/collection1/],
 dataDir=[null]
   [junit4]   2> 1214687 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX 
Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3975bc34
   [junit4]   2> 1214688 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.CachingDirectoryFactory return new directory for 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_31ADC7DE93344FFB-001/control-001/cores/collection1/data
   [junit4]   2> 1214688 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore New index directory detected: old=null 
new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_31ADC7DE93344FFB-001/control-001/cores/collection1/data/index/
   [junit4]   2> 1214688 WARN  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore [collection1] Solr index directory 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_31ADC7DE93344FFB-001/control-001/cores/collection1/data/index'
 doesn't exist. Creating new index...
   [junit4]   2> 1214688 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.CachingDirectoryFactory return new directory for 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_31ADC7DE93344FFB-001/control-001/cores/collection1/data/index
   [junit4]   2> 1214688 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.MockRandomMergePolicy: 
org.apache.lucene.index.MockRandomMergePolicy@b175625
   [junit4]   2> 1214689 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2>        commit{dir=MockDirectoryWrapper(RAMDirectory@3ad6b31 
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@56e6d73c),segFN=segments_1,generation=1}
   [junit4]   2> 1214689 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore newest commit generation = 1
   [junit4]   2> 1214689 INFO  
(OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:41759_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore 
Looking for old index directories to cleanup for core collection1 in 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_31ADC7DE93344FFB-001/control-001/cores/collection1/data/
   [junit4]   2> 1214689 WARN  
(OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:41759_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.DirectoryFactory 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_31ADC7DE93344FFB-001/control-001/cores/collection1/data/
 does not point to a valid data directory; skipping clean-up of old index 
directories.
   [junit4]   2> 1214692 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain 
"nodistrib"
   [junit4]   2> 1214692 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain 
"dedupe"
   [junit4]   2> 1214693 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.UpdateRequestProcessorChain inserting 
DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1214693 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain 
"stored_sig"
   [junit4]   2> 1214693 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.UpdateRequestProcessorChain inserting 
DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1214693 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain 
"distrib-dup-test-chain-explicit"
   [junit4]   2> 1214693 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain 
"distrib-dup-test-chain-implicit"
   [junit4]   2> 1214693 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.UpdateRequestProcessorChain inserting 
DistributedUpdateProcessorFactory into updateRequestProcessorChain 
"distrib-dup-test-chain-implicit"
   [junit4]   2> 1214693 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating 
implicit default
   [junit4]   2> 1214695 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1214696 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1214697 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1214698 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1214701 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.RequestHandlers Registered paths: 
/admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
   [junit4]   2> 1214701 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore Using default statsCache cache: 
org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1214702 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.UpdateHandler Using UpdateLog implementation: 
org.apache.solr.update.UpdateLog
   [junit4]   2> 1214702 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1214703 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore Hard AutoCommit: disabled
   [junit4]   2> 1214703 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore Soft AutoCommit: disabled
   [junit4]   2> 1214703 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: 
maxMergeAtOnce=39, maxMergeAtOnceExplicit=15, maxMergedSegmentMB=63.7119140625, 
floorSegmentMB=1.392578125, forceMergeDeletesPctAllowed=22.274349037346344, 
segmentsPerTier=30.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1214703 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2>        commit{dir=MockDirectoryWrapper(RAMDirectory@3ad6b31 
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@56e6d73c),segFN=segments_1,generation=1}
   [junit4]   2> 1214703 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore newest commit generation = 1
   [junit4]   2> 1214703 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.s.SolrIndexSearcher Opening Searcher@1bb80d83[collection1] main
   [junit4]   2> 1214704 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.c.ZkStateReader Load collection config 
from:/collections/control_collection
   [junit4]   2> 1214705 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.c.ZkStateReader path=/collections/control_collection configName=conf1 
specified config exists in ZooKeeper
   [junit4]   2> 1214705 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the 
RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1214706 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf1
   [junit4]   2> 1214706 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 1214706 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1214706 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.r.ManagedResourceStorage No data found for znode 
/configs/conf1/_rest_managed.json
   [junit4]   2> 1214706 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1214706 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 1214707 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1214707 INFO  
(searcherExecutor-3052-thread-1-processing-n:127.0.0.1:41759_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41759_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore 
[collection1] Registered new searcher Searcher@1bb80d83[collection1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1214707 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 1214707 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536 version 
buckets from index
   [junit4]   2> 1214707 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket 
highest value from index
   [junit4]   2> 1214707 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.UpdateLog Could not find max version in index or recent updates, using 
new clock 1514473794894299136
   [junit4]   2> 1214709 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.UpdateLog Took 2.0ms to seed version buckets with highest version 
1514473794894299136
   [junit4]   2> 1214709 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 1214710 INFO  
(coreLoadExecutor-3051-thread-1-processing-n:127.0.0.1:41759_) 
[n:127.0.0.1:41759_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 1214710 INFO  
(coreZkRegister-3045-thread-1-processing-n:127.0.0.1:41759_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41759_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController 
Register replica - core:collection1 address:http://127.0.0.1:41759 
collection:control_collection shard:shard1
   [junit4]   2> 1214711 INFO  
(coreZkRegister-3045-thread-1-processing-n:127.0.0.1:41759_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41759_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.c.SolrZkClient makePath: 
/collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1214713 INFO  
(coreZkRegister-3045-thread-1-processing-n:127.0.0.1:41759_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41759_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector 
Joined leadership election with path: 
/collections/control_collection/leader_elect/shard1/election/94654612068892676-core_node1-n_0000000000
   [junit4]   2> 1214714 INFO  
(coreZkRegister-3045-thread-1-processing-n:127.0.0.1:41759_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41759_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.ShardLeaderElectionContext Running the leader process for shard shard1
   [junit4]   2> 1214715 INFO  
(coreZkRegister-3045-thread-1-processing-n:127.0.0.1:41759_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41759_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1214715 INFO  
(coreZkRegister-3045-thread-1-processing-n:127.0.0.1:41759_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41759_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1214715 INFO  
(coreZkRegister-3045-thread-1-processing-n:127.0.0.1:41759_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41759_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy 
Sync replicas to http://127.0.0.1:41759/collection1/
   [junit4]   2> 1214715 INFO  
(coreZkRegister-3045-thread-1-processing-n:127.0.0.1:41759_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41759_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy 
Sync Success - now sync replicas to me
   [junit4]   2> 1214715 INFO  
(coreZkRegister-3045-thread-1-processing-n:127.0.0.1:41759_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41759_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy 
http://127.0.0.1:41759/collection1/ has no replicas
   [junit4]   2> 1214715 INFO  
(coreZkRegister-3045-thread-1-processing-n:127.0.0.1:41759_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41759_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.ShardLeaderElectionContext I am the new leader: 
http://127.0.0.1:41759/collection1/ shard1
   [junit4]   2> 1214715 INFO  
(OverseerStateUpdate-94654612068892676-127.0.0.1:41759_-n_0000000000) 
[n:127.0.0.1:41759_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message 
= {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 1214716 INFO  
(coreZkRegister-3045-thread-1-processing-n:127.0.0.1:41759_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41759_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1214718 INFO  
(coreZkRegister-3045-thread-1-processing-n:127.0.0.1:41759_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41759_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
   [junit4]   2> 1214720 INFO  
(OverseerStateUpdate-94654612068892676-127.0.0.1:41759_-n_0000000000) 
[n:127.0.0.1:41759_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message 
= {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:41759";,
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 1
   [junit4]   2> 1214822 INFO  
(zkCallback-1206-thread-1-processing-n:127.0.0.1:41759_) [n:127.0.0.1:41759_    
] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent 
state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred 
- updating... (live nodes size: 1)
   [junit4]   2> 1214871 INFO  
(coreZkRegister-3045-thread-1-processing-n:127.0.0.1:41759_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41759_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController 
We are http://127.0.0.1:41759/collection1/ and leader is 
http://127.0.0.1:41759/collection1/
   [junit4]   2> 1214871 INFO  
(coreZkRegister-3045-thread-1-processing-n:127.0.0.1:41759_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41759_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController 
No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:41759
   [junit4]   2> 1214871 INFO  
(coreZkRegister-3045-thread-1-processing-n:127.0.0.1:41759_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41759_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController 
I am the leader, no recovery necessary
   [junit4]   2> 1214871 INFO  
(coreZkRegister-3045-thread-1-processing-n:127.0.0.1:41759_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41759_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController 
publishing state=active
   [junit4]   2> 1214871 INFO  
(coreZkRegister-3045-thread-1-processing-n:127.0.0.1:41759_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41759_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController 
numShards not found on descriptor - reading it from system property
   [junit4]   2> 1214873 INFO  
(OverseerStateUpdate-94654612068892676-127.0.0.1:41759_-n_0000000000) 
[n:127.0.0.1:41759_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message 
= {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:41759";,
   [junit4]   2>   "node_name":"127.0.0.1:41759_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 2
   [junit4]   2> 1214873 INFO  
(OverseerStateUpdate-94654612068892676-127.0.0.1:41759_-n_0000000000) 
[n:127.0.0.1:41759_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 
message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:41759";,
   [junit4]   2>   "node_name":"127.0.0.1:41759_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1214976 INFO  
(zkCallback-1206-thread-1-processing-n:127.0.0.1:41759_) [n:127.0.0.1:41759_    
] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent 
state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred 
- updating... (live nodes size: 1)
   [junit4]   2> 1215019 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1215020 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1215020 INFO  (zkCallback-1208-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@29f69bf3 
name:ZooKeeperConnection Watcher:127.0.0.1:41939/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1215021 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1215021 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1215021 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1215023 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause 
connection loss:false
   [junit4]   2> 1215329 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_31ADC7DE93344FFB-001/shard-1-001/cores/collection1
   [junit4]   2> 1215330 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_31ADC7DE93344FFB-001/shard-1-001
   [junit4]   2> 1215331 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.e.j.s.Server jetty-9.2.13.v20150730
   [junit4]   2> 1215332 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@70971522{/,null,AVAILABLE}
   [junit4]   2> 1215333 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.e.j.s.ServerConnector Started 
ServerConnector@769d189b{HTTP/1.1}{127.0.0.1:55153}
   [junit4]   2> 1215333 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.e.j.s.Server Started @1218651ms
   [junit4]   2> 1215333 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_31ADC7DE93344FFB-001/tempDir-001/jetty1,
 solrconfig=solrconfig.xml, hostContext=/, hostPort=55153, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_31ADC7DE93344FFB-001/shard-1-001/cores}
   [junit4]   2> 1215333 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): 
sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 1215333 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_31ADC7DE93344FFB-001/shard-1-001/'
   [junit4]   2> 1215348 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1215348 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1215353 INFO  (zkCallback-1209-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@289e05ab 
name:ZooKeeperConnection Watcher:127.0.0.1:41939/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1215353 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1215353 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1215354 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 1215354 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_31ADC7DE93344FFB-001/shard-1-001/solr.xml
   [junit4]   2> 1215359 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.CoresLocator Config-defined core root directory: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_31ADC7DE93344FFB-001/shard-1-001/cores
   [junit4]   2> 1215359 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.CoreContainer New CoreContainer 1709262308
   [junit4]   2> 1215359 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer 
[instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_31ADC7DE93344FFB-001/shard-1-001/]
   [junit4]   2> 1215359 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.CoreContainer loading shared library: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_31ADC7DE93344FFB-001/shard-1-001/lib
   [junit4]   2> 1215359 WARN  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.SolrResourceLoader Can't find (or read) directory to add to 
classloader: lib (resolved as: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_31ADC7DE93344FFB-001/shard-1-001/lib).
   [junit4]   2> 1215366 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 
90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 
20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 
2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : 
false,useRetries : false,
   [junit4]   2> 1215367 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with 
params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1215368 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1215368 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j 
(org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1215368 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41939/solr
   [junit4]   2> 1215368 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1215368 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1215368 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1215370 INFO  (zkCallback-1211-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@63966300 
name:ZooKeeperConnection Watcher:127.0.0.1:41939 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1215370 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1215370 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1215371 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:55153_    ] o.a.s.c.c.ConnectionManager Waiting for client to 
connect to ZooKeeper
   [junit4]   2> 1215372 INFO  
(zkCallback-1212-thread-1-processing-n:127.0.0.1:55153_) [n:127.0.0.1:55153_    
] o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@3c6be338 
name:ZooKeeperConnection Watcher:127.0.0.1:41939/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1215372 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:55153_    ] o.a.s.c.c.ConnectionManager Client is connected to 
ZooKeeper
   [junit4]   2> 1215379 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:55153_    ] o.a.s.c.c.ZkStateReader Updating cluster state from 
ZooKeeper... 
   [junit4]   2> 1216384 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:55153_    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:55153_
   [junit4]   2> 1216385 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:55153_    ] o.a.s.c.c.SolrZkClient makePath: 
/live_nodes/127.0.0.1:55153_
   [junit4]   2> 1216386 INFO  (zkCallback-1208-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected 
type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live 
nodes size: 1)
   [junit4]   2> 1216386 INFO  
(zkCallback-1206-thread-1-processing-n:127.0.0.1:41759_) [n:127.0.0.1:41759_    
] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected 
type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live 
nodes size: 1)
   [junit4]   2> 1216386 INFO  
(zkCallback-1212-thread-1-processing-n:127.0.0.1:55153_) [n:127.0.0.1:55153_    
] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected 
type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live 
nodes size: 1)
   [junit4]   2> 1216387 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:55153_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1216388 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:55153_    ] o.a.s.c.LeaderElector Joined leadership election with 
path: /overseer_elect/election/94654612068892680-127.0.0.1:55153_-n_0000000001
   [junit4]   2> 1216389 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:55153_    ] o.a.s.c.LeaderElector Watching path 
/overseer_elect/election/94654612068892676-127.0.0.1:41759_-n_0000000000 to 
know if I could be the leader
   [junit4]   2> 1216398 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:55153_    ] o.a.s.c.CoreContainer Security conf doesn't exist. 
Skipping setup for authorization module.
   [junit4]   2> 1216398 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:55153_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1216399 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:55153_    ] o.a.s.c.CoresLocator Looking for core definitions 
underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_31ADC7DE93344FFB-001/shard-1-001/cores
   [junit4]   2> 1216400 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:55153_    ] o.a.s.c.SolrCore Created CoreDescriptor: 
{name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, 
loadOnStartup=true, configSetProperties=configsetprops.json, 
instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_31ADC7DE93344FFB-001/shard-1-001/cores/collection1,
 collection=collection1, 
absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_31ADC7DE93344FFB-001/shard-1-001/cores/collection1/,
 coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 1216400 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:55153_    ] o.a.s.c.CoresLocator Found core collection1 in 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_31ADC7DE93344FFB-001/shard-1-001/cores/collection1/
   [junit4]   2> 1216400 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:55153_    ] o.a.s.c.CoresLocator Found 1 core definitions
   [junit4]   2> 1216401 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:55153_    ] o.a.s.s.SolrDispatchFilter 
user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2
   [junit4]   2> 1216401 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[31ADC7DE93344FFB])
 [n:127.0.0.1:55153_    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() 
done
   [junit4]   2> 1216401 INFO  
(coreLoadExecutor-3062-thread-1-processing-n:127.0.0.1:55153_) 
[n:127.0.0.1:55153_ c:collection1   x:collection1] o.a.s.c.ZkController 
publishing state=down
   [junit4]   2> 1216401 INFO  
(coreLoadExecutor-3062-thread-1-processing-n:127.0.0.1:55153_) 
[n:127.0.0.1:55153_ c:collection1   x:collection1] o.a.s.c.ZkController 
numShards not found on descriptor - reading it from system property
   [junit4]   2> 1216403 INFO  
(coreLoadExecutor-3062-thread-1-processing-n:127.0.0.1:55153_) 
[n:127.0.0.1:55153_ c:collection1   x:collection1] o.a.s.c.ZkController look 
for our core node name
   [junit4]   2> 1216403 INFO  
(OverseerStateUpdate-94654612068892676-127.0.0.1:41759_-n_0000000000) 
[n:127.0.0.1:41759_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message 
= {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles

[...truncated too long message...]

e.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 3919073 ERROR 
(zkCallback-1956-thread-11-processing-n:127.0.0.1:55989_-EventThread) 
[n:127.0.0.1:55989_    ] o.a.z.ClientCnxn Error while calling watcher 
   [junit4]   2> java.util.concurrent.RejectedExecutionException: Task 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1@693cddee 
rejected from 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@45f4e6bc[Terminated,
 pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 213]
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:214)
   [junit4]   2>        at 
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
   [junit4]   2>        at 
org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:266)
   [junit4]   2>        at 
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2>        at 
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 3919073 ERROR 
(zkCallback-1956-thread-11-processing-n:127.0.0.1:55989_-EventThread) 
[n:127.0.0.1:55989_    ] o.a.z.ClientCnxn Error while calling watcher 
   [junit4]   2> java.util.concurrent.RejectedExecutionException: Task 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1@128f3cf4 
rejected from 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@45f4e6bc[Terminated,
 pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 213]
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:214)
   [junit4]   2>        at 
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
   [junit4]   2>        at 
org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:266)
   [junit4]   2>        at 
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2>        at 
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 3919073 ERROR 
(zkCallback-1956-thread-11-processing-n:127.0.0.1:55989_-EventThread) 
[n:127.0.0.1:55989_    ] o.a.z.ClientCnxn Error while calling watcher 
   [junit4]   2> java.util.concurrent.RejectedExecutionException: Task 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1@7c8f462d 
rejected from 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@45f4e6bc[Terminated,
 pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 213]
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:214)
   [junit4]   2>        at 
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
   [junit4]   2>        at 
org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:266)
   [junit4]   2>        at 
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2>        at 
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 3919074 ERROR 
(zkCallback-1956-thread-11-processing-n:127.0.0.1:55989_-EventThread) 
[n:127.0.0.1:55989_    ] o.a.z.ClientCnxn Error while calling watcher 
   [junit4]   2> java.util.concurrent.RejectedExecutionException: Task 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1@748a28a0 
rejected from 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@45f4e6bc[Terminated,
 pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 213]
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:214)
   [junit4]   2>        at 
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
   [junit4]   2>        at 
org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:266)
   [junit4]   2>        at 
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2>        at 
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 3919074 ERROR 
(zkCallback-1956-thread-11-processing-n:127.0.0.1:55989_-EventThread) 
[n:127.0.0.1:55989_    ] o.a.z.ClientCnxn Error while calling watcher 
   [junit4]   2> java.util.concurrent.RejectedExecutionException: Task 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1@183b50f0 
rejected from 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@45f4e6bc[Terminated,
 pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 213]
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:214)
   [junit4]   2>        at 
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
   [junit4]   2>        at 
org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:266)
   [junit4]   2>        at 
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2>        at 
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> NOTE: test params are: codec=CheapBastard, 
sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=ja_JP, 
timezone=Pacific/Efate
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 
1.8.0_45 (64-bit)/cpus=4,threads=1,free=235241968,total=535298048
   [junit4]   2> NOTE: All tests run in this JVM: 
[UniqFieldsUpdateProcessorFactoryTest, CollectionTooManyReplicasTest, 
TestReqParamsAPI, ZkSolrClientTest, SortByFunctionTest, 
DistribDocExpirationUpdateProcessorTest, TestInitQParser, MinimalSchemaTest, 
HdfsNNFailoverTest, TestObjectReleaseTracker, ZkStateWriterTest, 
SoftAutoCommitTest, TestSchemaManager, SolrCloudExampleTest, CircularListTest, 
TestClusterStateMutator, HdfsBasicDistributedZk2Test, TestTrie, 
PreAnalyzedFieldTest, SpellCheckComponentTest, 
TestExclusionRuleCollectionAccess, TestSolrDeletionPolicy2, 
TriLevelCompositeIdRoutingTest, AddSchemaFieldsUpdateProcessorFactoryTest, 
CdcrRequestHandlerTest, TestChildDocTransformer, ZkCLITest, 
ChaosMonkeySafeLeaderTest, UpdateRequestProcessorFactoryTest, TestUpdate, 
DistributedExpandComponentTest, CdcrReplicationDistributedZkTest, 
DocValuesTest, SpatialFilterTest, BasicFunctionalityTest, SolrPluginUtilsTest, 
CollectionsAPIDistributedZkTest]
   [junit4] Completed [383/547] on J1 in 2645.39s, 1 test, 1 error <<< FAILURES!

[...truncated 518 lines...]
   [junit4] JVM J1: stdout was not empty, see: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/temp/junit4-J1-20151008_101222_978.sysout
   [junit4] >>> JVM J1: stdout (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/heapdumps/java_pid13726.hprof
 ...
   [junit4] Heap dump file created [681108533 bytes in 6.184 secs]
   [junit4] <<< JVM J1: EOF ----

[...truncated 12 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:782:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:719:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:59:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:233:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:516:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1432:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:987:
 There were test failures: 547 suites (8 ignored), 2148 tests, 2 errors, 1 
failure, 29 ignored (17 assumptions) [seed: 31ADC7DE93344FFB]

Total time: 192 minutes 40 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to