Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-7.x/61/

11 tests failed.
FAILED:  org.apache.solr.cloud.MoveReplicaHDFSTest.testFailedMove

Error Message:
No live SolrServers available to handle this 
request:[http://127.0.0.1:46031/solr/MoveReplicaHDFSTest_failed_coll_true, 
http://127.0.0.1:37179/solr/MoveReplicaHDFSTest_failed_coll_true]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available 
to handle this 
request:[http://127.0.0.1:46031/solr/MoveReplicaHDFSTest_failed_coll_true, 
http://127.0.0.1:37179/solr/MoveReplicaHDFSTest_failed_coll_true]
        at 
__randomizedtesting.SeedInfo.seed([8448C61E6428EE4D:2E8515ECD3FB3B9D]:0)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:462)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1106)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:886)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:993)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:819)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
        at 
org.apache.solr.cloud.MoveReplicaTest.testFailedMove(MoveReplicaTest.java:288)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: 
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at 
http://127.0.0.1:46031/solr/MoveReplicaHDFSTest_failed_coll_true: no servers 
hosting shard: shard2
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:436)
        ... 46 more


FAILED:  org.apache.solr.cloud.MultiThreadedOCPTest.test

Error Message:


Stack Trace:
java.lang.AssertionError
        at 
__randomizedtesting.SeedInfo.seed([8448C61E6428EE4D:C1CF9C4CAD483B5]:0)
        at org.junit.Assert.fail(Assert.java:92)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertTrue(Assert.java:54)
        at 
org.apache.solr.cloud.MultiThreadedOCPTest.testFillWorkQueue(MultiThreadedOCPTest.java:113)
        at 
org.apache.solr.cloud.MultiThreadedOCPTest.test(MultiThreadedOCPTest.java:68)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest.testInactiveShardCleanup

Error Message:
should be at least one inactive event

Stack Trace:
java.lang.AssertionError: should be at least one inactive event
        at 
__randomizedtesting.SeedInfo.seed([8448C61E6428EE4D:9964066C056BC946]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at 
org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest.testInactiveShardCleanup(ScheduledMaintenanceTriggerTest.java:224)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.cloud.autoscaling.SearchRateTriggerIntegrationTest.testBelowSearchRate

Error Message:
[Op{action=DELETEREPLICA, hints={COLL_SHARD=[{   
"first":"belowRate_collection",   "second":"shard1"}], REPLICA=[core_node6]}}, 
Op{action=DELETEREPLICA, hints={COLL_SHARD=[{   "first":"belowRate_collection", 
  "second":"shard1"}], REPLICA=[core_node8]}}, Op{action=DELETEREPLICA, 
hints={COLL_SHARD=[{   "first":"belowRate_collection",   "second":"shard1"}], 
REPLICA=[core_node10]}}, Op{action=NONE, 
hints={SRC_NODE=[127.0.0.1:37749_solr]}}, Op{action=NONE, 
hints={SRC_NODE=[127.0.0.1:35431_solr]}}, Op{action=NONE, 
hints={SRC_NODE=[127.0.0.1:38327_solr]}}, Op{action=NONE, 
hints={SRC_NODE=[127.0.0.1:45021_solr]}}, Op{action=NONE, 
hints={SRC_NODE=[127.0.0.1:34118_solr]}}] expected:<7> but was:<8>

Stack Trace:
java.lang.AssertionError: [Op{action=DELETEREPLICA, hints={COLL_SHARD=[{
  "first":"belowRate_collection",
  "second":"shard1"}], REPLICA=[core_node6]}}, Op{action=DELETEREPLICA, 
hints={COLL_SHARD=[{
  "first":"belowRate_collection",
  "second":"shard1"}], REPLICA=[core_node8]}}, Op{action=DELETEREPLICA, 
hints={COLL_SHARD=[{
  "first":"belowRate_collection",
  "second":"shard1"}], REPLICA=[core_node10]}}, Op{action=NONE, 
hints={SRC_NODE=[127.0.0.1:37749_solr]}}, Op{action=NONE, 
hints={SRC_NODE=[127.0.0.1:35431_solr]}}, Op{action=NONE, 
hints={SRC_NODE=[127.0.0.1:38327_solr]}}, Op{action=NONE, 
hints={SRC_NODE=[127.0.0.1:45021_solr]}}, Op{action=NONE, 
hints={SRC_NODE=[127.0.0.1:34118_solr]}}] expected:<7> but was:<8>
        at 
__randomizedtesting.SeedInfo.seed([8448C61E6428EE4D:EEE5607E3CC06B5E]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at 
org.apache.solr.cloud.autoscaling.SearchRateTriggerIntegrationTest.testBelowSearchRate(SearchRateTriggerIntegrationTest.java:394)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testNodeLost

Error Message:
last state: DocCollection(testNodeLost//clusterstate.json/261)={   
"replicationFactor":"10",   "pullReplicas":"0",   
"router":{"name":"compositeId"},   "maxShardsPerNode":"5",   
"autoAddReplicas":"false",   "nrtReplicas":"10",   "tlogReplicas":"0",   
"autoCreated":"true",   "shards":{     "shard20":{       "replicas":{         
"core_node191":{           "core":"testNodeLost_shard20_replica_n191",          
 "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,     
      "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10081_solr",      
     "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node200":{           
"core":"testNodeLost_shard20_replica_n200",           "leader":"true",          
 "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,     
      "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10015_solr",      
     "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node199":{           
"core":"testNodeLost_shard20_replica_n199",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10121_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node198":{           
"core":"testNodeLost_shard20_replica_n198",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10117_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node197":{           
"core":"testNodeLost_shard20_replica_n197",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10037_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node196":{           
"core":"testNodeLost_shard20_replica_n196",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10101_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node195":{           
"core":"testNodeLost_shard20_replica_n195",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10016_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node194":{           
"core":"testNodeLost_shard20_replica_n194",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10026_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node193":{           
"core":"testNodeLost_shard20_replica_n193",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10098_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node192":{           
"core":"testNodeLost_shard20_replica_n192",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10126_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"73330000-7fffffff",       
"state":"active"},     "shard10":{       "replicas":{         "core_node92":{   
        "core":"testNodeLost_shard10_replica_n92",           "leader":"true",   
        "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10126_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node91":{ 
          "core":"testNodeLost_shard10_replica_n91",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10081_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node94":{           
"core":"testNodeLost_shard10_replica_n94",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10026_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node93":{           
"core":"testNodeLost_shard10_replica_n93",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10098_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node96":{           
"core":"testNodeLost_shard10_replica_n96",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10101_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node95":{           
"core":"testNodeLost_shard10_replica_n95",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10016_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node98":{           
"core":"testNodeLost_shard10_replica_n98",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10117_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node100":{           
"core":"testNodeLost_shard10_replica_n100",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10015_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node97":{           
"core":"testNodeLost_shard10_replica_n97",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10037_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node99":{           
"core":"testNodeLost_shard10_replica_n99",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10121_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"f3330000-ffffffff",       
"state":"active"},     "shard11":{       "replicas":{         "core_node108":{  
         "core":"testNodeLost_shard11_replica_n108",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10084_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node107":{           
"core":"testNodeLost_shard11_replica_n107",           "leader":"true",          
 "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,     
      "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10059_solr",      
     "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node106":{           
"core":"testNodeLost_shard11_replica_n106",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10050_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node105":{           
"core":"testNodeLost_shard11_replica_n105",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10021_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node104":{           
"core":"testNodeLost_shard11_replica_n104",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10034_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node103":{           
"core":"testNodeLost_shard11_replica_n103",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10055_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node102":{           
"core":"testNodeLost_shard11_replica_n102",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10027_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node101":{           
"core":"testNodeLost_shard11_replica_n101",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10038_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node110":{           
"core":"testNodeLost_shard11_replica_n110",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10103_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node109":{           
"core":"testNodeLost_shard11_replica_n109",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10132_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"0-ccbffff",       
"state":"active"},     "shard12":{       "replicas":{         "core_node119":{  
         "core":"testNodeLost_shard12_replica_n119",           "leader":"true", 
          "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10024_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         
"core_node118":{           "core":"testNodeLost_shard12_replica_n118",          
 "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,     
      "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10106_solr",      
     "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node117":{           
"core":"testNodeLost_shard12_replica_n117",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10102_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node116":{           
"core":"testNodeLost_shard12_replica_n116",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10096_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node115":{           
"core":"testNodeLost_shard12_replica_n115",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10046_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node114":{           
"core":"testNodeLost_shard12_replica_n114",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10020_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node113":{           
"core":"testNodeLost_shard12_replica_n113",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10052_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node112":{           
"core":"testNodeLost_shard12_replica_n112",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10072_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node111":{           
"core":"testNodeLost_shard12_replica_n111",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10041_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node120":{           
"core":"testNodeLost_shard12_replica_n120",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10012_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"ccc0000-1998ffff",       
"state":"active"},     "shard13":{       "replicas":{         "core_node129":{  
         "core":"testNodeLost_shard13_replica_n129",           "leader":"true", 
          "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10053_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         
"core_node206":{           "node_name":"127.0.0.1:10103_solr",           
"core":"testNodeLost_shard13_replica_n205",           "state":"active",         
  "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node128":{      
     "core":"testNodeLost_shard13_replica_n128",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10099_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node126":{           
"core":"testNodeLost_shard13_replica_n126",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10062_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node125":{           
"core":"testNodeLost_shard13_replica_n125",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10093_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node124":{           
"core":"testNodeLost_shard13_replica_n124",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10087_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node123":{           
"core":"testNodeLost_shard13_replica_n123",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10115_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node122":{           
"core":"testNodeLost_shard13_replica_n122",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10073_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node121":{           
"core":"testNodeLost_shard13_replica_n121",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10045_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node130":{           
"core":"testNodeLost_shard13_replica_n130",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10029_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"19990000-2665ffff",       
"state":"active"},     "shard2":{       "replicas":{         "core_node18":{    
       "core":"testNodeLost_shard2_replica_n18",           "leader":"true",     
      "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10106_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node17":{ 
          "core":"testNodeLost_shard2_replica_n17",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10102_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node19":{           
"core":"testNodeLost_shard2_replica_n19",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10024_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node20":{           
"core":"testNodeLost_shard2_replica_n20",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10012_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node12":{           
"core":"testNodeLost_shard2_replica_n12",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10072_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node11":{           
"core":"testNodeLost_shard2_replica_n11",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10041_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node14":{           
"core":"testNodeLost_shard2_replica_n14",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10020_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node13":{           
"core":"testNodeLost_shard2_replica_n13",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10052_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node16":{           
"core":"testNodeLost_shard2_replica_n16",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10096_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node15":{           
"core":"testNodeLost_shard2_replica_n15",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10046_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"8ccc0000-9998ffff",       
"state":"active"},     "shard3":{       "replicas":{         "core_node29":{    
       "core":"testNodeLost_shard3_replica_n29",           "leader":"true",     
      "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10053_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node28":{ 
          "core":"testNodeLost_shard3_replica_n28",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10099_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node30":{           
"core":"testNodeLost_shard3_replica_n30",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10029_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node21":{           
"core":"testNodeLost_shard3_replica_n21",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10045_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node23":{           
"core":"testNodeLost_shard3_replica_n23",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10115_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node22":{           
"core":"testNodeLost_shard3_replica_n22",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10073_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node25":{           
"core":"testNodeLost_shard3_replica_n25",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10093_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node24":{           
"core":"testNodeLost_shard3_replica_n24",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10087_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node26":{           
"core":"testNodeLost_shard3_replica_n26",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10062_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node208":{           
"node_name":"127.0.0.1:10059_solr",           
"core":"testNodeLost_shard3_replica_n207",           "state":"active",          
 "INDEX.sizeInBytes":1,           "type":"NRT"}},       
"range":"99990000-a665ffff",       "state":"active"},     "shard14":{       
"replicas":{         "core_node139":{           
"core":"testNodeLost_shard14_replica_n139",           "leader":"true",          
 "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,     
      "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10090_solr",      
     "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node138":{           
"core":"testNodeLost_shard14_replica_n138",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10127_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node137":{           
"core":"testNodeLost_shard14_replica_n137",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10110_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node136":{           
"core":"testNodeLost_shard14_replica_n136",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10067_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node135":{           
"core":"testNodeLost_shard14_replica_n135",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10129_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node134":{           
"core":"testNodeLost_shard14_replica_n134",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10100_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node133":{           
"core":"testNodeLost_shard14_replica_n133",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10119_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node132":{           
"core":"testNodeLost_shard14_replica_n132",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10056_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node131":{           
"core":"testNodeLost_shard14_replica_n131",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10018_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node140":{           
"core":"testNodeLost_shard14_replica_n140",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10085_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"26660000-3332ffff",       
"state":"active"},     "shard15":{       "replicas":{         "core_node149":{  
         "core":"testNodeLost_shard15_replica_n149",           "leader":"true", 
          "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10086_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         
"core_node148":{           "core":"testNodeLost_shard15_replica_n148",          
 "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,     
      "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10048_solr",      
     "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node147":{           
"core":"testNodeLost_shard15_replica_n147",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10133_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node146":{           
"core":"testNodeLost_shard15_replica_n146",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10008_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node145":{           
"core":"testNodeLost_shard15_replica_n145",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10064_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node144":{           
"core":"testNodeLost_shard15_replica_n144",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10116_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node143":{           
"core":"testNodeLost_shard15_replica_n143",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10094_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node142":{           
"core":"testNodeLost_shard15_replica_n142",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10014_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node141":{           
"core":"testNodeLost_shard15_replica_n141",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10006_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node150":{           
"core":"testNodeLost_shard15_replica_n150",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10078_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"33330000-3fffffff",       
"state":"active"},     "shard4":{       "replicas":{         "core_node39":{    
       "core":"testNodeLost_shard4_replica_n39",           "leader":"true",     
      "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10090_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node40":{ 
          "core":"testNodeLost_shard4_replica_n40",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10085_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node32":{           
"core":"testNodeLost_shard4_replica_n32",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10056_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node31":{           
"core":"testNodeLost_shard4_replica_n31",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10018_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node34":{           
"core":"testNodeLost_shard4_replica_n34",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10100_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node33":{           
"core":"testNodeLost_shard4_replica_n33",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10119_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node36":{           
"core":"testNodeLost_shard4_replica_n36",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10067_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node35":{           
"core":"testNodeLost_shard4_replica_n35",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10129_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node38":{           
"core":"testNodeLost_shard4_replica_n38",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10127_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node37":{           
"core":"testNodeLost_shard4_replica_n37",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10110_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"a6660000-b332ffff",       
"state":"active"},     "shard5":{       "replicas":{         "core_node50":{    
       "core":"testNodeLost_shard5_replica_n50",           "leader":"true",     
      "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10078_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node41":{ 
          "core":"testNodeLost_shard5_replica_n41",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10006_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node43":{           
"core":"testNodeLost_shard5_replica_n43",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10094_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node42":{           
"core":"testNodeLost_shard5_replica_n42",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10014_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node45":{           
"core":"testNodeLost_shard5_replica_n45",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10064_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node44":{           
"core":"testNodeLost_shard5_replica_n44",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10116_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node47":{           
"core":"testNodeLost_shard5_replica_n47",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10133_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node46":{           
"core":"testNodeLost_shard5_replica_n46",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10008_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node49":{           
"core":"testNodeLost_shard5_replica_n49",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10086_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node48":{           
"core":"testNodeLost_shard5_replica_n48",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10048_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"b3330000-bfffffff",       
"state":"active"},     "shard16":{       "replicas":{         "core_node159":{  
         "core":"testNodeLost_shard16_replica_n159",           "leader":"true", 
          "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10131_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         
"core_node158":{           "core":"testNodeLost_shard16_replica_n158",          
 "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,     
      "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10031_solr",      
     "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node157":{           
"core":"testNodeLost_shard16_replica_n157",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10105_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node156":{           
"core":"testNodeLost_shard16_replica_n156",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10079_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node155":{           
"core":"testNodeLost_shard16_replica_n155",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10030_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node154":{           
"core":"testNodeLost_shard16_replica_n154",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10123_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node153":{           
"core":"testNodeLost_shard16_replica_n153",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10120_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node152":{           
"core":"testNodeLost_shard16_replica_n152",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10066_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node151":{           
"core":"testNodeLost_shard16_replica_n151",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10122_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node160":{           
"core":"testNodeLost_shard16_replica_n160",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10023_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"40000000-4ccbffff",       
"state":"active"},     "shard17":{       "replicas":{         "core_node169":{  
         "core":"testNodeLost_shard17_replica_n169",           "leader":"true", 
          "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10039_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         
"core_node168":{           "core":"testNodeLost_shard17_replica_n168",          
 "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,     
      "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10019_solr",      
     "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node167":{           
"core":"testNodeLost_shard17_replica_n167",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10111_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node166":{           
"core":"testNodeLost_shard17_replica_n166",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10080_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node165":{           
"core":"testNodeLost_shard17_replica_n165",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10125_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node164":{           
"core":"testNodeLost_shard17_replica_n164",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10047_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node163":{           
"core":"testNodeLost_shard17_replica_n163",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10033_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node162":{           
"core":"testNodeLost_shard17_replica_n162",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10044_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node161":{           
"core":"testNodeLost_shard17_replica_n161",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10051_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node170":{           
"core":"testNodeLost_shard17_replica_n170",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10091_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"4ccc0000-5998ffff",       
"state":"active"},     "shard18":{       "replicas":{         "core_node180":{  
         "core":"testNodeLost_shard18_replica_n180",           "leader":"true", 
          "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10077_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         
"core_node179":{           "core":"testNodeLost_shard18_replica_n179",          
 "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,     
      "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10040_solr",      
     "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node178":{           
"core":"testNodeLost_shard18_replica_n178",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10068_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node177":{           
"core":"testNodeLost_shard18_replica_n177",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10097_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node176":{           
"core":"testNodeLost_shard18_replica_n176",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10108_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node175":{           
"core":"testNodeLost_shard18_replica_n175",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10010_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node174":{           
"core":"testNodeLost_shard18_replica_n174",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10060_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node173":{           
"core":"testNodeLost_shard18_replica_n173",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10036_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node172":{           
"core":"testNodeLost_shard18_replica_n172",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10112_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node171":{           
"core":"testNodeLost_shard18_replica_n171",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10092_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"59990000-6665ffff",       
"state":"active"},     "shard19":{       "replicas":{         "core_node190":{  
         "core":"testNodeLost_shard19_replica_n190",           "leader":"true", 
          "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10042_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         
"core_node202":{           "node_name":"127.0.0.1:10072_solr",           
"core":"testNodeLost_shard19_replica_n201",           "state":"active",         
  "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node189":{      
     "core":"testNodeLost_shard19_replica_n189",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10075_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node188":{           
"core":"testNodeLost_shard19_replica_n188",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10058_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node187":{           
"core":"testNodeLost_shard19_replica_n187",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10114_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node186":{           
"core":"testNodeLost_shard19_replica_n186",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10011_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node184":{           
"core":"testNodeLost_shard19_replica_n184",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10035_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node183":{           
"core":"testNodeLost_shard19_replica_n183",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10007_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node182":{           
"core":"testNodeLost_shard19_replica_n182",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10017_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node181":{           
"core":"testNodeLost_shard19_replica_n181",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10061_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"66660000-7332ffff",       
"state":"active"},     "shard1":{       "replicas":{         "core_node1":{     
      "core":"testNodeLost_shard1_replica_n1",           "leader":"true",       
    "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,  
         "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10038_solr",   
        "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node6":{           
"core":"testNodeLost_shard1_replica_n6",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10050_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node10":{           
"core":"testNodeLost_shard1_replica_n10",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10103_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node7":{           
"core":"testNodeLost_shard1_replica_n7",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10059_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node8":{           
"core":"testNodeLost_shard1_replica_n8",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10084_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node9":{           
"core":"testNodeLost_shard1_replica_n9",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10132_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node2":{           
"core":"testNodeLost_shard1_replica_n2",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10027_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node3":{           
"core":"testNodeLost_shard1_replica_n3",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10055_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node4":{           
"core":"testNodeLost_shard1_replica_n4",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10034_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node5":{           
"core":"testNodeLost_shard1_replica_n5",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10021_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"80000000-8ccbffff",       
"state":"active"},     "shard6":{       "replicas":{         "core_node60":{    
       "core":"testNodeLost_shard6_replica_n60",           "leader":"true",     
      "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10023_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node52":{ 
          "core":"testNodeLost_shard6_replica_n52",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10066_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node51":{           
"core":"testNodeLost_shard6_replica_n51",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10122_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node54":{           
"core":"testNodeLost_shard6_replica_n54",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10123_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node53":{           
"core":"testNodeLost_shard6_replica_n53",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10120_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node56":{           
"core":"testNodeLost_shard6_replica_n56",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10079_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node55":{           
"core":"testNodeLost_shard6_replica_n55",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10030_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node58":{           
"core":"testNodeLost_shard6_replica_n58",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10031_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node57":{           
"core":"testNodeLost_shard6_replica_n57",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10105_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node59":{           
"core":"testNodeLost_shard6_replica_n59",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10131_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"c0000000-cccbffff",       
"state":"active"},     "shard7":{       "replicas":{         "core_node70":{    
       "core":"testNodeLost_shard7_replica_n70",           "leader":"true",     
      "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10091_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node61":{ 
          "core":"testNodeLost_shard7_replica_n61",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10051_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node63":{           
"core":"testNodeLost_shard7_replica_n63",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10033_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node62":{           
"core":"testNodeLost_shard7_replica_n62",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10044_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node65":{           
"core":"testNodeLost_shard7_replica_n65",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10125_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node64":{           
"core":"testNodeLost_shard7_replica_n64",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10047_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node67":{           
"core":"testNodeLost_shard7_replica_n67",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10111_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node66":{           
"core":"testNodeLost_shard7_replica_n66",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10080_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node69":{           
"core":"testNodeLost_shard7_replica_n69",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10039_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node68":{           
"core":"testNodeLost_shard7_replica_n68",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10019_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"cccc0000-d998ffff",       
"state":"active"},     "shard8":{       "replicas":{         "core_node80":{    
       "core":"testNodeLost_shard8_replica_n80",           "leader":"true",     
      "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10077_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node72":{ 
          "core":"testNodeLost_shard8_replica_n72",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10112_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node71":{           
"core":"testNodeLost_shard8_replica_n71",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10092_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node74":{           
"core":"testNodeLost_shard8_replica_n74",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10060_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node73":{           
"core":"testNodeLost_shard8_replica_n73",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10036_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node76":{           
"core":"testNodeLost_shard8_replica_n76",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10108_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node75":{           
"core":"testNodeLost_shard8_replica_n75",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10010_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node78":{           
"core":"testNodeLost_shard8_replica_n78",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10068_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node77":{           
"core":"testNodeLost_shard8_replica_n77",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10097_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node79":{           
"core":"testNodeLost_shard8_replica_n79",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10040_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"d9990000-e665ffff",       
"state":"active"},     "shard9":{       "replicas":{         "core_node90":{    
       "core":"testNodeLost_shard9_replica_n90",           "leader":"true",     
      "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10042_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node81":{ 
          "core":"testNodeLost_shard9_replica_n81",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10061_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node204":{           
"node_name":"127.0.0.1:10056_solr",           
"core":"testNodeLost_shard9_replica_n203",           "state":"active",          
 "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node83":{        
   "core":"testNodeLost_shard9_replica_n83",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10007_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node82":{           
"core":"testNodeLost_shard9_replica_n82",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10017_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node84":{           
"core":"testNodeLost_shard9_replica_n84",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10035_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node87":{           
"core":"testNodeLost_shard9_replica_n87",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10114_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node86":{           
"core":"testNodeLost_shard9_replica_n86",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10011_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node89":{           
"core":"testNodeLost_shard9_replica_n89",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10075_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node88":{           
"core":"testNodeLost_shard9_replica_n88",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10058_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"e6660000-f332ffff",       
"state":"active"}}}

Stack Trace:
java.util.concurrent.TimeoutException: last state: 
DocCollection(testNodeLost//clusterstate.json/261)={
  "replicationFactor":"10",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"5",
  "autoAddReplicas":"

[...truncated too long message...]

figure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/null699932580
     [copy] Copying 39 files to 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/null699932580
   [delete] Deleting directory 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/null699932580

resolve-example:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

resolve-server:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/null454020974
     [copy] Copying 246 files to 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/null454020974
   [delete] Deleting directory 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/null454020974

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: 
org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath]         confs: [default]
[ivy:cachepath]         found 
org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath]         found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath]         found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath]         found org.apache.httpcomponents#httpclient;4.3.6 in 
public
[ivy:cachepath]         found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath]         found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath]         found commons-codec#commons-codec;1.6 in public
[ivy:cachepath]         found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 57ms :: artifacts dl 1ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for 
further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 308 minutes 37 seconds
Archiving artifacts
WARN: No artifacts found that match the file pattern 
"**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
WARN: java.lang.InterruptedException: no matches found within 10000
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Reply via email to