Build: https://builds.apache.org/job/Lucene-Solr-BadApples-NightlyTests-7.x/31/

15 tests failed.
FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([ED0D86404D2DC638]:0)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Error Message:
Captured an uncaught exception in thread: Thread[id=3335, 
name=SolrRrdBackendFactory-1253-thread-1, state=RUNNABLE, 
group=TGRP-FullSolrCloudDistribCmdsTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=3335, name=SolrRrdBackendFactory-1253-thread-1, 
state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space


FAILED:  
org.apache.solr.cloud.autoscaling.sim.TestSimComputePlanAction.testNodeWithMultipleReplicasLost

Error Message:
ComputePlanAction should have computed exactly 2 operation expected:<2> but 
was:<3>

Stack Trace:
java.lang.AssertionError: ComputePlanAction should have computed exactly 2 
operation expected:<2> but was:<3>
        at 
__randomizedtesting.SeedInfo.seed([ED0D86404D2DC638:DDCD67C2C55F2764]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at 
org.apache.solr.cloud.autoscaling.sim.TestSimComputePlanAction.testNodeWithMultipleReplicasLost(TestSimComputePlanAction.java:246)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.CollectionsAPISolrJTest.testCollectionProp

Error Message:
Error from server at https://127.0.0.1:41681/solr: Error setting property for 
collection collectionPropTest

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at https://127.0.0.1:41681/solr: Error setting property for 
collection collectionPropTest
        at 
__randomizedtesting.SeedInfo.seed([ED0D86404D2DC638:8FDE83ED5B656EC1]:0)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
        at 
org.apache.solr.cloud.CollectionsAPISolrJTest.testCollectionProp(CollectionsAPISolrJTest.java:539)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.CollectionsAPISolrJTest.testAddAndDeleteReplica

Error Message:
Error from server at https://127.0.0.1:41681/solr: Could not find collection : 
solrj_replicatests

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at https://127.0.0.1:41681/solr: Could not find collection : 
solrj_replicatests
        at 
__randomizedtesting.SeedInfo.seed([ED0D86404D2DC638:3BDFF154AE1D182]:0)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
        at 
org.apache.solr.cloud.CollectionsAPISolrJTest.testAddAndDeleteReplica(CollectionsAPISolrJTest.java:479)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.CollectionsAPISolrJTest.testSplitShard

Error Message:
Error from server at https://127.0.0.1:41681/solr: Could not find collection : 
solrj_test_splitshard

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at https://127.0.0.1:41681/solr: Could not find collection : 
solrj_test_splitshard
        at 
__randomizedtesting.SeedInfo.seed([ED0D86404D2DC638:36072B2C53D8FA87]:0)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
        at 
org.apache.solr.cloud.CollectionsAPISolrJTest.testSplitShard(CollectionsAPISolrJTest.java:400)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.cloud.CollectionsAPISolrJTest.testCreateWithDefaultConfigSet

Error Message:
expected:<4> but was:<1>

Stack Trace:
java.lang.AssertionError: expected:<4> but was:<1>
        at 
__randomizedtesting.SeedInfo.seed([ED0D86404D2DC638:A3A627071292BCCB]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at 
org.apache.solr.cloud.CollectionsAPISolrJTest.testCreateWithDefaultConfigSet(CollectionsAPISolrJTest.java:96)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([ED0D86404D2DC638]:0)


FAILED:  
org.apache.solr.cloud.autoscaling.SearchRateTriggerIntegrationTest.testBelowSearchRate

Error Message:
[Op{action=DELETEREPLICA, hints={COLL_SHARD=[{   
"first":"belowRate_collection",   "second":"shard1"}], REPLICA=[core_node10]}}, 
Op{action=DELETEREPLICA, hints={COLL_SHARD=[{   "first":"belowRate_collection", 
  "second":"shard1"}], REPLICA=[core_node8]}}, Op{action=DELETEREPLICA, 
hints={COLL_SHARD=[{   "first":"belowRate_collection",   "second":"shard1"}], 
REPLICA=[core_node6]}}, Op{action=NONE, 
hints={SRC_NODE=[127.0.0.1:35258_solr]}}, Op{action=NONE, 
hints={SRC_NODE=[127.0.0.1:34392_solr]}}, Op{action=NONE, 
hints={SRC_NODE=[127.0.0.1:40663_solr]}}, Op{action=NONE, 
hints={SRC_NODE=[127.0.0.1:37332_solr]}}, Op{action=NONE, 
hints={SRC_NODE=[127.0.0.1:37654_solr]}}] expected:<7> but was:<8>

Stack Trace:
java.lang.AssertionError: [Op{action=DELETEREPLICA, hints={COLL_SHARD=[{
  "first":"belowRate_collection",
  "second":"shard1"}], REPLICA=[core_node10]}}, Op{action=DELETEREPLICA, 
hints={COLL_SHARD=[{
  "first":"belowRate_collection",
  "second":"shard1"}], REPLICA=[core_node8]}}, Op{action=DELETEREPLICA, 
hints={COLL_SHARD=[{
  "first":"belowRate_collection",
  "second":"shard1"}], REPLICA=[core_node6]}}, Op{action=NONE, 
hints={SRC_NODE=[127.0.0.1:35258_solr]}}, Op{action=NONE, 
hints={SRC_NODE=[127.0.0.1:34392_solr]}}, Op{action=NONE, 
hints={SRC_NODE=[127.0.0.1:40663_solr]}}, Op{action=NONE, 
hints={SRC_NODE=[127.0.0.1:37332_solr]}}, Op{action=NONE, 
hints={SRC_NODE=[127.0.0.1:37654_solr]}}] expected:<7> but was:<8>
        at 
__randomizedtesting.SeedInfo.seed([ED0D86404D2DC638:87A0202015C5432B]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at 
org.apache.solr.cloud.autoscaling.SearchRateTriggerIntegrationTest.testBelowSearchRate(SearchRateTriggerIntegrationTest.java:400)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.cloud.autoscaling.SearchRateTriggerIntegrationTest.testDeleteNode

Error Message:
unexpected DELETENODE status: 
{responseHeader={status=0,QTime=73},status={state=notfound,msg=Did not find 
[search_rate_trigger3/4e64c6751e4284T42ckoyf03h6hpwby6dr3udoer/0] in any tasks 
queue}}

Stack Trace:
java.lang.AssertionError: unexpected DELETENODE status: 
{responseHeader={status=0,QTime=73},status={state=notfound,msg=Did not find 
[search_rate_trigger3/4e64c6751e4284T42ckoyf03h6hpwby6dr3udoer/0] in any tasks 
queue}}
        at 
__randomizedtesting.SeedInfo.seed([ED0D86404D2DC638:CF9F48C27AE74945]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at 
org.apache.solr.cloud.autoscaling.SearchRateTriggerIntegrationTest.lambda$testDeleteNode$5(SearchRateTriggerIntegrationTest.java:693)
        at java.util.ArrayList.forEach(ArrayList.java:1257)
        at 
org.apache.solr.cloud.autoscaling.SearchRateTriggerIntegrationTest.testDeleteNode(SearchRateTriggerIntegrationTest.java:677)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.testNodeLost

Error Message:
last ClusterState: znodeVersion: 646 live nodes:[127.0.0.1:10043_solr, 
127.0.0.1:10057_solr, 127.0.0.1:10112_solr, 127.0.0.1:10012_solr, 
127.0.0.1:10060_solr, 127.0.0.1:10073_solr, 127.0.0.1:10058_solr, 
127.0.0.1:10013_solr, 127.0.0.1:10090_solr, 127.0.0.1:10088_solr, 
127.0.0.1:10027_solr, 127.0.0.1:10039_solr, 127.0.0.1:10025_solr, 
127.0.0.1:10044_solr, 127.0.0.1:10036_solr, 127.0.0.1:10028_solr, 
127.0.0.1:10092_solr, 127.0.0.1:10052_solr, 127.0.0.1:10006_solr, 
127.0.0.1:10079_solr, 127.0.0.1:10041_solr, 127.0.0.1:10087_solr, 
127.0.0.1:10009_solr, 127.0.0.1:10098_solr, 127.0.0.1:10030_solr, 
127.0.0.1:10014_solr, 127.0.0.1:10011_solr, 127.0.0.1:10040_solr, 
127.0.0.1:10085_solr, 127.0.0.1:10099_solr, 127.0.0.1:10101_solr, 
127.0.0.1:10046_solr, 127.0.0.1:10068_solr, 127.0.0.1:10071_solr, 
127.0.0.1:10063_solr, 127.0.0.1:10023_solr, 127.0.0.1:10007_solr, 
127.0.0.1:10107_solr, 127.0.0.1:10015_solr, 127.0.0.1:10055_solr, 
127.0.0.1:10069_solr, 127.0.0.1:10110_solr, 127.0.0.1:10010_solr, 
127.0.0.1:10016_solr, 127.0.0.1:10070_solr, 127.0.0.1:10062_solr, 
127.0.0.1:10077_solr, 127.0.0.1:10038_solr, 127.0.0.1:10032_solr, 
127.0.0.1:10108_solr, 127.0.0.1:10008_solr, 127.0.0.1:10024_solr, 
127.0.0.1:10093_solr, 127.0.0.1:10061_solr, 127.0.0.1:10067_solr, 
127.0.0.1:10050_solr, 127.0.0.1:10075_solr, 127.0.0.1:10078_solr, 
127.0.0.1:10086_solr, 127.0.0.1:10089_solr, 127.0.0.1:10097_solr, 
127.0.0.1:10031_solr, 127.0.0.1:10034_solr, 127.0.0.1:10091_solr, 
127.0.0.1:10094_solr, 127.0.0.1:10106_solr, 127.0.0.1:10100_solr, 
127.0.0.1:10042_solr, 127.0.0.1:10037_solr, 127.0.0.1:10109_solr, 
127.0.0.1:10029_solr, 127.0.0.1:10083_solr, 127.0.0.1:10053_solr, 
127.0.0.1:10059_solr, 127.0.0.1:10048_solr, 127.0.0.1:10103_solr, 
127.0.0.1:10064_solr, 127.0.0.1:10111_solr, 127.0.0.1:10056_solr, 
127.0.0.1:10072_solr] 
collections:{.system=DocCollection(.system//clusterstate.json/645)={   
"replicationFactor":"3",   "pullReplicas":"0",   
"router":{"name":"compositeId"},   "maxShardsPerNode":"1",   
"autoAddReplicas":"false",   "nrtReplicas":"1",   "tlogReplicas":"0",   
"autoCreated":"true",   "shards":{"shard1":{       "replicas":{         
"core_node7":{           "core":".system_shard1_replica_n1",           
"leader":"true",           "SEARCHER.searcher.maxDoc":179,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":99740,         
  "node_name":"127.0.0.1:10072_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.289011359214783E-5,           
"SEARCHER.searcher.numDocs":179},         "core_node8":{           
"core":".system_shard1_replica_n2",           "SEARCHER.searcher.maxDoc":179,   
        "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":99740, 
          "node_name":"127.0.0.1:10056_solr",           "state":"active",       
    "type":"NRT",           "INDEX.sizeInGB":9.289011359214783E-5,           
"SEARCHER.searcher.numDocs":179},         "core_node9":{           
"core":".system_shard1_replica_n3",           "SEARCHER.searcher.maxDoc":179,   
        "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":99740, 
          "node_name":"127.0.0.1:10103_solr",           "state":"active",       
    "type":"NRT",           "INDEX.sizeInGB":9.289011359214783E-5,           
"SEARCHER.searcher.numDocs":179}},       "range":"80000000-7fffffff",       
"state":"active"}}}, 
testNodeLost=DocCollection(testNodeLost//clusterstate.json/645)={   
"replicationFactor":"1",   "pullReplicas":"0",   
"router":{"name":"compositeId"},   "maxShardsPerNode":"5",   
"autoAddReplicas":"false",   "nrtReplicas":"10",   "tlogReplicas":"0",   
"autoCreated":"true",   "shards":{     "shard20":{       "replicas":{         
"core_node191":{           "core":"testNodeLost_shard20_replica_n191",          
 "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,     
      "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10037_solr",  
         "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node200":{           "core":"testNodeLost_shard20_replica_n200",   
        "leader":"true",           "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,         
  "node_name":"127.0.0.1:10072_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node199":{           
"core":"testNodeLost_shard20_replica_n199",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10056_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node198":{           "core":"testNodeLost_shard20_replica_n198",   
        "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,         
  "node_name":"127.0.0.1:10103_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node197":{           
"core":"testNodeLost_shard20_replica_n197",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10103_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node196":{           "core":"testNodeLost_shard20_replica_n196",   
        "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,         
  "node_name":"127.0.0.1:10056_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node195":{           
"core":"testNodeLost_shard20_replica_n195",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10072_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node194":{           "core":"testNodeLost_shard20_replica_n194",   
        "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,         
  "node_name":"127.0.0.1:10059_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node193":{           
"core":"testNodeLost_shard20_replica_n193",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10053_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node192":{           "core":"testNodeLost_shard20_replica_n192",   
        "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,         
  "node_name":"127.0.0.1:10109_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0}},       "range":"73330000-7fffffff",       
"state":"active"},     "shard10":{       "replicas":{         "core_node218":{  
         "node_name":"127.0.0.1:10034_solr",           
"core":"testNodeLost_shard10_replica_n217",           "state":"active",         
  "INDEX.sizeInBytes":10240,           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6},         "core_node206":{           
"node_name":"127.0.0.1:10109_solr",           
"core":"testNodeLost_shard10_replica_n205",           "state":"active",         
  "INDEX.sizeInBytes":10240,           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6},         "core_node226":{           
"node_name":"127.0.0.1:10075_solr",           
"core":"testNodeLost_shard10_replica_n225",           "state":"active",         
  "INDEX.sizeInBytes":10240,           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6},         "core_node204":{           
"node_name":"127.0.0.1:10053_solr",           
"core":"testNodeLost_shard10_replica_n203",           "state":"active",         
  "INDEX.sizeInBytes":10240,           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6},         "core_node224":{           
"node_name":"127.0.0.1:10078_solr",           
"core":"testNodeLost_shard10_replica_n223",           "state":"active",         
  "INDEX.sizeInBytes":10240,           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6},         "core_node202":{           
"leader":"true",           "core":"testNodeLost_shard10_replica_n201",          
 "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10059_solr",       
    "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6},         "core_node244":{           
"node_name":"127.0.0.1:10068_solr",           
"core":"testNodeLost_shard10_replica_n243",           "state":"active",         
  "INDEX.sizeInBytes":10240,           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6},         "core_node220":{           
"node_name":"127.0.0.1:10031_solr",           
"core":"testNodeLost_shard10_replica_n219",           "state":"active",         
  "INDEX.sizeInBytes":10240,           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6},         "core_node252":{           
"node_name":"127.0.0.1:10011_solr",           
"core":"testNodeLost_shard10_replica_n251",           "state":"active",         
  "INDEX.sizeInBytes":10240,           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6},         "core_node208":{           
"node_name":"127.0.0.1:10037_solr",           
"core":"testNodeLost_shard10_replica_n207",           "state":"active",         
  "INDEX.sizeInBytes":10240,           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6}},       "range":"f3330000-ffffffff",       
"state":"active"},     "shard11":{       "replicas":{         "core_node228":{  
         "node_name":"127.0.0.1:10050_solr",           
"core":"testNodeLost_shard11_replica_n227",           "state":"active",         
  "INDEX.sizeInBytes":10240,           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6},         "core_node248":{           
"node_name":"127.0.0.1:10085_solr",           
"core":"testNodeLost_shard11_replica_n247",           "state":"active",         
  "INDEX.sizeInBytes":10240,           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6},         "core_node256":{           
"node_name":"127.0.0.1:10030_solr",           
"core":"testNodeLost_shard11_replica_n255",           "state":"active",         
  "INDEX.sizeInBytes":10240,           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6},         "core_node234":{           
"node_name":"127.0.0.1:10062_solr",           
"core":"testNodeLost_shard11_replica_n233",           "state":"active",         
  "INDEX.sizeInBytes":10240,           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6},         "core_node212":{           
"node_name":"127.0.0.1:10106_solr",           
"core":"testNodeLost_shard11_replica_n211",           "state":"active",         
  "INDEX.sizeInBytes":10240,           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6},         "core_node222":{           
"node_name":"127.0.0.1:10097_solr",           
"core":"testNodeLost_shard11_replica_n221",           "state":"active",         
  "INDEX.sizeInBytes":10240,           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6},         "core_node210":{           
"leader":"true",           "core":"testNodeLost_shard11_replica_n209",          
 "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10100_solr",       
    "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6},         "core_node242":{           
"node_name":"127.0.0.1:10063_solr",           
"core":"testNodeLost_shard11_replica_n241",           "state":"active",         
  "INDEX.sizeInBytes":10240,           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6},         "core_node230":{           
"node_name":"127.0.0.1:10024_solr",           
"core":"testNodeLost_shard11_replica_n229",           "state":"active",         
  "INDEX.sizeInBytes":10240,           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6},         "core_node250":{           
"node_name":"127.0.0.1:10040_solr",           
"core":"testNodeLost_shard11_replica_n249",           "state":"active",         
  "INDEX.sizeInBytes":10240,           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6}},       "range":"0-ccbffff",       
"state":"active"},     "shard12":{       "replicas":{         "core_node119":{  
         "core":"testNodeLost_shard12_replica_n119",           "leader":"true", 
          "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,         
  "node_name":"127.0.0.1:10052_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node118":{           
"core":"testNodeLost_shard12_replica_n118",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10036_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node115":{           "core":"testNodeLost_shard12_replica_n115",   
        "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,         
  "node_name":"127.0.0.1:10095_solr",           "state":"down",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node236":{           
"node_name":"127.0.0.1:10110_solr",           
"core":"testNodeLost_shard12_replica_n235",           "state":"active",         
  "INDEX.sizeInBytes":10240,           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6},         "core_node114":{           
"core":"testNodeLost_shard12_replica_n114",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10102_solr",   
        "state":"down",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node268":{           "node_name":"127.0.0.1:10044_solr",           
"core":"testNodeLost_shard12_replica_n267",           "state":"active",         
  "INDEX.sizeInBytes":10240,           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6},         "core_node120":{           
"core":"testNodeLost_shard12_replica_n120",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10006_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node262":{           "node_name":"127.0.0.1:10041_solr",           
"core":"testNodeLost_shard12_replica_n261",           "state":"active",         
  "INDEX.sizeInBytes":10240,           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6},         "core_node260":{           
"node_name":"127.0.0.1:10087_solr",           
"core":"testNodeLost_shard12_replica_n259",           "state":"active",         
  "INDEX.sizeInBytes":10240,           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6},         "core_node270":{           
"node_name":"127.0.0.1:10025_solr",           
"core":"testNodeLost_shard12_replica_n269",           "state":"active",         
  "INDEX.sizeInBytes":10240,           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6}},       "range":"ccc0000-1998ffff",       
"state":"active"},     "shard2":{       "replicas":{         "core_node18":{    
       "core":"testNodeLost_shard2_replica_n18",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10110_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node17":{           "core":"testNodeLost_shard2_replica_n17",      
     "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0, 
          "INDEX.sizeInBytes":10240,           
"node_name":"127.0.0.1:10062_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node19":{           
"core":"testNodeLost_shard2_replica_n19",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10069_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node20":{           "core":"testNodeLost_shard2_replica_n20",      
     "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0, 
          "INDEX.sizeInBytes":10240,           
"node_name":"127.0.0.1:10015_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node12":{           
"core":"testNodeLost_shard2_replica_n12",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10078_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node11":{           "core":"testNodeLost_shard2_replica_n11",      
     "leader":"true",           "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,         
  "node_name":"127.0.0.1:10097_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node14":{           
"core":"testNodeLost_shard2_replica_n14",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10050_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node13":{           "core":"testNodeLost_shard2_replica_n13",      
     "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0, 
          "INDEX.sizeInBytes":10240,           
"node_name":"127.0.0.1:10075_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node16":{           
"core":"testNodeLost_shard2_replica_n16",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10008_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node15":{           "core":"testNodeLost_shard2_replica_n15",      
     "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0, 
          "INDEX.sizeInBytes":10240,           
"node_name":"127.0.0.1:10024_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0}},       "range":"8ccc0000-9998ffff",       
"state":"active"},     "shard13":{       "replicas":{         "core_node129":{  
         "core":"testNodeLost_shard13_replica_n129",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10055_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node128":{           "core":"testNodeLost_shard13_replica_n128",   
        "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,         
  "node_name":"127.0.0.1:10107_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node127":{           
"core":"testNodeLost_shard13_replica_n127",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10007_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node126":{           "core":"testNodeLost_shard13_replica_n126",   
        "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,         
  "node_name":"127.0.0.1:10023_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node125":{           
"core":"testNodeLost_shard13_replica_n125",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10071_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node124":{           "core":"testNodeLost_shard13_replica_n124",   
        "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,         
  "node_name":"127.0.0.1:10101_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node123":{           
"core":"testNodeLost_shard13_replica_n123",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10099_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node122":{           "core":"testNodeLost_shard13_replica_n122",   
        "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,         
  "node_name":"127.0.0.1:10098_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node121":{           
"core":"testNodeLost_shard13_replica_n121",           "leader":"true",          
 "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,     
      "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10079_solr",  
         "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node130":{           "core":"testNodeLost_shard13_replica_n130",   
        "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,         
  "node_name":"127.0.0.1:10010_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0}},       "range":"19990000-2665ffff",       
"state":"active"},     "shard3":{       "replicas":{         "core_node29":{    
       "core":"testNodeLost_shard3_replica_n29",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10009_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node28":{           "core":"testNodeLost_shard3_replica_n28",      
     "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0, 
          "INDEX.sizeInBytes":10240,           
"node_name":"127.0.0.1:10030_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node30":{           
"core":"testNodeLost_shard3_replica_n30",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10087_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node21":{           "core":"testNodeLost_shard3_replica_n21",      
     "leader":"true",           "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,         
  "node_name":"127.0.0.1:10063_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node23":{           
"core":"testNodeLost_shard3_replica_n23",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10046_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node22":{           "core":"testNodeLost_shard3_replica_n22",      
     "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0, 
          "INDEX.sizeInBytes":10240,           
"node_name":"127.0.0.1:10068_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node25":{           
"core":"testNodeLost_shard3_replica_n25",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10040_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node24":{           "core":"testNodeLost_shard3_replica_n24",      
     "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0, 
          "INDEX.sizeInBytes":10240,           
"node_name":"127.0.0.1:10085_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node27":{           
"core":"testNodeLost_shard3_replica_n27",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10014_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node26":{           "core":"testNodeLost_shard3_replica_n26",      
     "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0, 
          "INDEX.sizeInBytes":10240,           
"node_name":"127.0.0.1:10011_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0}},       "range":"99990000-a665ffff",       
"state":"active"},     "shard14":{       "replicas":{         "core_node139":{  
         "core":"testNodeLost_shard14_replica_n139",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10067_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node138":{           "core":"testNodeLost_shard14_replica_n138",   
        "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,         
  "node_name":"127.0.0.1:10061_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node137":{           
"core":"testNodeLost_shard14_replica_n137",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10093_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node136":{           "core":"testNodeLost_shard14_replica_n136",   
        "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,         
  "node_name":"127.0.0.1:10108_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node135":{           
"core":"testNodeLost_shard14_replica_n135",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10032_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node134":{           "core":"testNodeLost_shard14_replica_n134",   
        "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,         
  "node_name":"127.0.0.1:10038_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node133":{           
"core":"testNodeLost_shard14_replica_n133",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10077_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node132":{           "core":"testNodeLost_shard14_replica_n132",   
        "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,         
  "node_name":"127.0.0.1:10070_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node131":{           
"core":"testNodeLost_shard14_replica_n131",           "leader":"true",          
 "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,     
      "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10016_solr",  
         "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node140":{           "core":"testNodeLost_shard14_replica_n140",   
        "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,         
  "node_name":"127.0.0.1:10086_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0}},       "range":"26660000-3332ffff",       
"state":"active"},     "shard4":{       "replicas":{         "core_node39":{    
       "core":"testNodeLost_shard4_replica_n39",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10090_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node40":{           "core":"testNodeLost_shard4_replica_n40",      
     "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0, 
          "INDEX.sizeInBytes":10240,           
"node_name":"127.0.0.1:10013_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node32":{           
"core":"testNodeLost_shard4_replica_n32",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10092_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node31":{           "core":"testNodeLost_shard4_replica_n31",      
     "leader":"true",           "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,         
  "node_name":"127.0.0.1:10041_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node34":{           
"core":"testNodeLost_shard4_replica_n34",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10044_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node33":{           "core":"testNodeLost_shard4_replica_n33",      
     "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0, 
          "INDEX.sizeInBytes":10240,           
"node_name":"127.0.0.1:10028_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node36":{           
"core":"testNodeLost_shard4_replica_n36",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10039_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node35":{           "core":"testNodeLost_shard4_replica_n35",      
     "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0, 
          "INDEX.sizeInBytes":10240,           
"node_name":"127.0.0.1:10025_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node38":{           
"core":"testNodeLost_shard4_replica_n38",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10088_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node37":{           "core":"testNodeLost_shard4_replica_n37",      
     "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0, 
          "INDEX.sizeInBytes":10240,           
"node_name":"127.0.0.1:10027_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0}},       "range":"a6660000-b332ffff",       
"state":"active"},     "shard15":{       "replicas":{         "core_node149":{  
         "core":"testNodeLost_shard15_replica_n149",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10057_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node148":{           "core":"testNodeLost_shard15_replica_n148",   
        "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,         
  "node_name":"127.0.0.1:10043_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node147":{           
"core":"testNodeLost_shard15_replica_n147",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10111_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node146":{           "core":"testNodeLost_shard15_replica_n146",   
        "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,         
  "node_name":"127.0.0.1:10064_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node145":{           
"core":"testNodeLost_shard15_replica_n145",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10048_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node144":{           "core":"testNodeLost_shard15_replica_n144",   
        "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,         
  "node_name":"127.0.0.1:10083_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node143":{           
"core":"testNodeLost_shard15_replica_n143",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10029_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node142":{           "core":"testNodeLost_shard15_replica_n142",   
        "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,         
  "node_name":"127.0.0.1:10042_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node141":{           
"core":"testNodeLost_shard15_replica_n141",           "leader":"true",          
 "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,     
      "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10089_solr",  
         "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node150":{           "core":"testNodeLost_shard15_replica_n150",   
        "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,         
  "node_name":"127.0.0.1:10112_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0}},       "range":"33330000-3fffffff",       
"state":"active"},     "shard16":{       "replicas":{         "core_node159":{  
         "core":"testNodeLost_shard16_replica_n159",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10039_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node158":{           "core":"testNodeLost_shard16_replica_n158",   
        "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,         
  "node_name":"127.0.0.1:10027_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node157":{           
"core":"testNodeLost_shard16_replica_n157",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10088_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node156":{           "core":"testNodeLost_shard16_replica_n156",   
        "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,         
  "node_name":"127.0.0.1:10090_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node155":{           
"core":"testNodeLost_shard16_replica_n155",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10013_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node154":{           "core":"testNodeLost_shard16_replica_n154",   
        "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,         
  "node_name":"127.0.0.1:10058_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node153":{           
"core":"testNodeLost_shard16_replica_n153",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10073_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node152":{           "core":"testNodeLost_shard16_replica_n152",   
        "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,         
  "node_name":"127.0.0.1:10060_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node151":{           
"core":"testNodeLost_shard16_replica_n151",           "leader":"true",          
 "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,     
      "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10012_solr",  
         "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node160":{           "core":"testNodeLost_shard16_replica_n160",   
        "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,         
  "node_name":"127.0.0.1:10025_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0}},       "range":"40000000-4ccbffff",       
"state":"active"},     "shard5":{       "replicas":{         "core_node50":{    
       "core":"testNodeLost_shard5_replica_n50",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10048_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node41":{           "core":"testNodeLost_shard5_replica_n41",      
     "leader":"true",           "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,         
  "node_name":"127.0.0.1:10058_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node43":{           
"core":"testNodeLost_shard5_replica_n43",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10060_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node42":{           "core":"testNodeLost_shard5_replica_n42",      
     "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0, 
          "INDEX.sizeInBytes":10240,           
"node_name":"127.0.0.1:10073_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node45":{           
"core":"testNodeLost_shard5_replica_n45",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10112_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node44":{           "core":"testNodeLost_shard5_replica_n44",      
     "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0, 
          "INDEX.sizeInBytes":10240,           
"node_name":"127.0.0.1:10012_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node47":{           
"core":"testNodeLost_shard5_replica_n47",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10043_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node46":{           "core":"testNodeLost_shard5_replica_n46",      
     "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0, 
          "INDEX.sizeInBytes":10240,           
"node_name":"127.0.0.1:10057_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node49":{           
"core":"testNodeLost_shard5_replica_n49",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10064_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node48":{           "core":"testNodeLost_shard5_replica_n48",      
     "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0, 
          "INDEX.sizeInBytes":10240,           
"node_name":"127.0.0.1:10111_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0}},       "range":"b3330000-bfffffff",       
"state":"active"},     "shard17":{       "replicas":{         "core_node169":{  
         "core":"testNodeLost_shard17_replica_n169",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10011_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node168":{           "core":"testNodeLost_shard17_replica_n168",   
        "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,         
  "node_name":"127.0.0.1:10014_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node167":{           
"core":"testNodeLost_shard17_replica_n167",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10030_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node166":{           "core":"testNodeLost_shard17_replica_n166",   
        "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,         
  "node_name":"127.0.0.1:10009_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node165":{           
"core":"testNodeLost_shard17_replica_n165",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10087_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node164":{           "core":"testNodeLost_shard17_replica_n164",   
        "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,         
  "node_name":"127.0.0.1:10041_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node163":{           
"core":"testNodeLost_shard17_replica_n163",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10092_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node162":{           "core":"testNodeLost_shard17_replica_n162",   
        "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,         
  "node_name":"127.0.0.1:10028_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node161":{           
"core":"testNodeLost_shard17_replica_n161",           "leader":"true",          
 "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,     
      "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10044_solr",  
         "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node170":{           "core":"testNodeLost_shard17_replica_n170",   
        "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,         
  "node_name":"127.0.0.1:10040_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0}},       "range":"4ccc0000-5998ffff",       
"state":"active"},     "shard18":{       "replicas":{         "core_node180":{  
         "core":"testNodeLost_shard18_replica_n180",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10024_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node179":{           "core":"testNodeLost_shard18_replica_n179",   
        "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,         
  "node_name":"127.0.0.1:10008_solr",           "state":"active",           
"type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           
"SEARCHER.searcher.numDocs":0},         "core_node178":{           
"core":"testNodeLost_shard18_replica_n178",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10062_solr",   
        "state":"active",           "type":"NRT",           
"INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},  
       "core_node177":{           "core":"testNodeLost_shard18_replica_n177",   
        

[...truncated too long message...]

ndlerWrapper.java:132)
   [junit4]   2>         at 
org.eclipse.jetty.server.Server.handle(Server.java:531)
   [junit4]   2>         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
   [junit4]   2>         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
   [junit4]   2>         at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
   [junit4]   2>         at 
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
   [junit4]   2>         at 
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
   [junit4]   2>         at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
   [junit4]   2>         at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
   [junit4]   2>         at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
   [junit4]   2>         at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
   [junit4]   2>         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
   [junit4]   2>         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
   [junit4]   2>         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>   62) Thread[id=3530, name=searcherExecutor-1283-thread-1, 
state=WAITING, group=TGRP-FullSolrCloudDistribCmdsTest]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]   2>         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]   2>         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2>         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   [junit4]   2>         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   [junit4]   2>         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>   63) Thread[id=3532, name=searcherExecutor-1282-thread-1, 
state=WAITING, group=TGRP-FullSolrCloudDistribCmdsTest]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]   2>         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]   2>         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2>         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   [junit4]   2>         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   [junit4]   2>         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>   64) Thread[id=3538, name=searcherExecutor-1296-thread-1, 
state=WAITING, group=TGRP-FullSolrCloudDistribCmdsTest]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]   2>         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]   2>         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2>         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   [junit4]   2>         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   [junit4]   2>         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>   65) Thread[id=3545, name=searcherExecutor-1298-thread-1, 
state=WAITING, group=TGRP-FullSolrCloudDistribCmdsTest]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]   2>         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]   2>         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2>         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   [junit4]   2>         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   [junit4]   2>         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>   66) Thread[id=3542, name=searcherExecutor-1295-thread-1, 
state=WAITING, group=TGRP-FullSolrCloudDistribCmdsTest]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]   2>         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]   2>         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2>         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   [junit4]   2>         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   [junit4]   2>         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: test params are: codec=Lucene70, 
sim=RandomSimilarity(queryNorm=true): {}, locale=et-EE, 
timezone=Pacific/Tongatapu
   [junit4]   2> NOTE: Linux 4.4.0-112-generic amd64/Oracle Corporation 
1.8.0_172 (64-bit)/cpus=4,threads=81,free=94586984,total=470810624
   [junit4]   2> NOTE: All tests run in this JVM: [EchoParamsTest, 
DistributedDebugComponentTest, BigEndianAscendingWordDeserializerTest, 
HDFSCollectionsAPITest, PeerSyncWithLeaderAndIndexFingerprintCachingTest, 
TestManagedSynonymFilterFactory, TestValueSourceCache, 
TestUnifiedSolrHighlighter, AlternateDirectoryTest, 
FieldMutatingUpdateProcessorTest, SolrMetricManagerTest, StatsComponentTest, 
TestBulkSchemaAPI, ScheduledTriggerIntegrationTest, TestSimNodeAddedTrigger, 
TestConfigsApi, RAMDirectoryFactoryTest, 
DistributedFacetSimpleRefinementLongTailTest, MetricTriggerIntegrationTest, 
HLLUtilTest, TestExclusionRuleCollectionAccess, LegacyCloudClusterPropTest, 
ExplicitHLLTest, AtomicUpdateProcessorFactoryTest, TestClusterProperties, 
TestLuceneMatchVersion, TestWithCollection, MultiTermTest, TestManagedResource, 
DistanceFunctionTest, SystemInfoHandlerTest, SchemaApiFailureTest, 
DateMathParserTest, TestUpdate, TestSafeXMLParsing, TestSolr4Spatial, 
TestRandomDVFaceting, TestSearcherReuse, NumberUtilsTest, TestGroupingSearch, 
TestSimExtremeIndexing, TestSolrCloudWithHadoopAuthPlugin, 
HdfsRecoverLeaseTest, FullSolrCloudDistribCmdsTest]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 
'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=FullSolrCloudDistribCmdsTest -Dtests.seed=ED0D86404D2DC638 
-Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true 
-Dtests.badapples=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/test-data/enwiki.random.lines.txt
 -Dtests.locale=et-EE -Dtests.timezone=Pacific/Tongatapu -Dtests.asserts=true 
-Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J0 | FullSolrCloudDistribCmdsTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 
7200000 msec).
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([ED0D86404D2DC638]:0)Throwable #2: 
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=3335, name=SolrRrdBackendFactory-1253-thread-1, 
state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
   [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4] Completed [741/834 (6!)] on J0 in 36180.60s, 1 test, 3 errors <<< 
FAILURES!

[...truncated 280 lines...]
   [junit4] JVM J0: stdout was not empty, see: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/temp/junit4-J0-20181016_214143_6214374395222165447334.sysout
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/heapdumps/java_pid14131.hprof
 ...
   [junit4] Heap dump file created [591149347 bytes in 57.966 secs]
   [junit4] <<< JVM J0: EOF ----

[...truncated 8858 lines...]
BUILD FAILED
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/build.xml:661:
 The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/build.xml:585:
 Some of the tests produced a heap dump, but did not fail. Maybe a suppressed 
OutOfMemoryError? Dumps created:
* java_pid14131.hprof

Total time: 975 minutes 13 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to