Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-master/53/

6 tests failed.
FAILED:  
org.apache.solr.cloud.api.collections.ShardSplitTest.testSplitAfterFailedSplit

Error Message:
expected:<1> but was:<2>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<2>
        at 
__randomizedtesting.SeedInfo.seed([7B3F86A0E05CE243:8272150FDC29AFC9]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at 
org.apache.solr.cloud.api.collections.ShardSplitTest.testSplitAfterFailedSplit(ShardSplitTest.java:284)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testListeners

Error Message:
both listeners should have fired expected:<2> but was:<0>

Stack Trace:
java.lang.AssertionError: both listeners should have fired expected:<2> but 
was:<0>
        at 
__randomizedtesting.SeedInfo.seed([7B3F86A0E05CE243:96FF11B05D307F9A]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at 
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testListeners(TestTriggerIntegration.java:978)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testNodeMarkersRegistration

Error Message:
Path /autoscaling/nodeAdded/127.0.0.1:10019_solr should have been deleted

Stack Trace:
java.lang.AssertionError: Path /autoscaling/nodeAdded/127.0.0.1:10019_solr 
should have been deleted
        at 
__randomizedtesting.SeedInfo.seed([7B3F86A0E05CE243:63850EACEE692FAC]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertFalse(Assert.java:68)
        at 
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testNodeMarkersRegistration(TestTriggerIntegration.java:863)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ZkControllerTest

Error Message:
ObjectTracker found 1 object(s) that were not released!!! [Overseer] 
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: 
org.apache.solr.cloud.Overseer  at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
  at org.apache.solr.cloud.Overseer.start(Overseer.java:545)  at 
org.apache.solr.cloud.OverseerElectionContext.runLeaderProcess(ElectionContext.java:851)
  at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170) 
 at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)  
at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)  at 
org.apache.solr.cloud.LeaderElector.retryElection(LeaderElector.java:393)  at 
org.apache.solr.cloud.ZkController.rejoinOverseerElection(ZkController.java:2081)
  at 
org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:331)
  at java.lang.Thread.run(Thread.java:748)  

Stack Trace:
java.lang.AssertionError: ObjectTracker found 1 object(s) that were not 
released!!! [Overseer]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: 
org.apache.solr.cloud.Overseer
        at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.cloud.Overseer.start(Overseer.java:545)
        at 
org.apache.solr.cloud.OverseerElectionContext.runLeaderProcess(ElectionContext.java:851)
        at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
        at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)
        at 
org.apache.solr.cloud.LeaderElector.retryElection(LeaderElector.java:393)
        at 
org.apache.solr.cloud.ZkController.rejoinOverseerElection(ZkController.java:2081)
        at 
org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:331)
        at java.lang.Thread.run(Thread.java:748)


        at __randomizedtesting.SeedInfo.seed([7B3F86A0E05CE243]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNull(Assert.java:551)
        at 
org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:303)
        at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:897)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ZkControllerTest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.ZkControllerTest:     
1) Thread[id=21183, 
name=OverseerAutoScalingTriggerThread-73773449468116995-127.0.0.1:8983_solr-n_0000000001,
 state=RUNNABLE, group=Overseer autoscaling triggers]         at 
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:329)         at 
java.lang.ClassLoader.loadClass(ClassLoader.java:357)         at 
org.apache.logging.log4j.core.impl.ThrowableProxy.loadClass(ThrowableProxy.java:613)
         at 
org.apache.logging.log4j.core.impl.ThrowableProxy.toExtendedStackTrace(ThrowableProxy.java:738)
         at 
org.apache.logging.log4j.core.impl.ThrowableProxy.<init>(ThrowableProxy.java:138)
         at 
org.apache.logging.log4j.core.impl.ThrowableProxy.<init>(ThrowableProxy.java:122)
         at 
org.apache.logging.log4j.core.impl.Log4jLogEvent.getThrownProxy(Log4jLogEvent.java:605)
         at 
org.apache.logging.log4j.core.pattern.ExtendedThrowablePatternConverter.format(ExtendedThrowablePatternConverter.java:64)
         at 
org.apache.logging.log4j.core.pattern.PatternFormatter.format(PatternFormatter.java:38)
         at 
org.apache.logging.log4j.core.layout.PatternLayout$PatternSerializer.toSerializable(PatternLayout.java:334)
         at 
org.apache.logging.log4j.core.layout.PatternLayout.toText(PatternLayout.java:233)
         at 
org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:218)
         at 
org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:58)
         at 
org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:177)
         at 
org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:170)
         at 
org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:161)
         at 
org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156)
         at 
org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129)
         at 
org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120)
         at 
org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
         at 
org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:448)
         at 
org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:433)
         at 
org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:417)    
     at 
org.apache.logging.log4j.core.config.LoggerConfig.logParent(LoggerConfig.java:439)
         at 
org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:434)
         at 
org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:417)    
     at 
org.apache.logging.log4j.core.config.LoggerConfig.logParent(LoggerConfig.java:439)
         at 
org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:434)
         at 
org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:417)    
     at 
org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:403)    
     at 
org.apache.logging.log4j.core.config.DefaultReliabilityStrategy.log(DefaultReliabilityStrategy.java:49)
         at org.apache.logging.log4j.core.Logger.logMessage(Logger.java:146)    
     at 
org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2170)
         at 
org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2125)
         at 
org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2108)
         at 
org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2002)
         at 
org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1974)
         at org.apache.logging.slf4j.Log4jLogger.error(Log4jLogger.java:319)    
     at 
org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:146)
         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE 
scope at org.apache.solr.cloud.ZkControllerTest: 
   1) Thread[id=21183, 
name=OverseerAutoScalingTriggerThread-73773449468116995-127.0.0.1:8983_solr-n_0000000001,
 state=RUNNABLE, group=Overseer autoscaling triggers]
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:329)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at 
org.apache.logging.log4j.core.impl.ThrowableProxy.loadClass(ThrowableProxy.java:613)
        at 
org.apache.logging.log4j.core.impl.ThrowableProxy.toExtendedStackTrace(ThrowableProxy.java:738)
        at 
org.apache.logging.log4j.core.impl.ThrowableProxy.<init>(ThrowableProxy.java:138)
        at 
org.apache.logging.log4j.core.impl.ThrowableProxy.<init>(ThrowableProxy.java:122)
        at 
org.apache.logging.log4j.core.impl.Log4jLogEvent.getThrownProxy(Log4jLogEvent.java:605)
        at 
org.apache.logging.log4j.core.pattern.ExtendedThrowablePatternConverter.format(ExtendedThrowablePatternConverter.java:64)
        at 
org.apache.logging.log4j.core.pattern.PatternFormatter.format(PatternFormatter.java:38)
        at 
org.apache.logging.log4j.core.layout.PatternLayout$PatternSerializer.toSerializable(PatternLayout.java:334)
        at 
org.apache.logging.log4j.core.layout.PatternLayout.toText(PatternLayout.java:233)
        at 
org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:218)
        at 
org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:58)
        at 
org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:177)
        at 
org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:170)
        at 
org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:161)
        at 
org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156)
        at 
org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129)
        at 
org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120)
        at 
org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
        at 
org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:448)
        at 
org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:433)
        at 
org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:417)
        at 
org.apache.logging.log4j.core.config.LoggerConfig.logParent(LoggerConfig.java:439)
        at 
org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:434)
        at 
org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:417)
        at 
org.apache.logging.log4j.core.config.LoggerConfig.logParent(LoggerConfig.java:439)
        at 
org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:434)
        at 
org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:417)
        at 
org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:403)
        at 
org.apache.logging.log4j.core.config.DefaultReliabilityStrategy.log(DefaultReliabilityStrategy.java:49)
        at org.apache.logging.log4j.core.Logger.logMessage(Logger.java:146)
        at 
org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2170)
        at 
org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2125)
        at 
org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2108)
        at 
org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2002)
        at 
org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1974)
        at org.apache.logging.slf4j.Log4jLogger.error(Log4jLogger.java:319)
        at 
org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:146)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([7B3F86A0E05CE243]:0)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testNodeLost

Error Message:
last state: DocCollection(testNodeLost//clusterstate.json/29)={   
"replicationFactor":"10",   "pullReplicas":"0",   
"router":{"name":"compositeId"},   "maxShardsPerNode":"5",   
"autoAddReplicas":"false",   "nrtReplicas":"10",   "tlogReplicas":"0",   
"autoCreated":"true",   "shards":{     "shard20":{       "replicas":{         
"core_node191":{           "core":"testNodeLost_shard20_replica_n191",          
 "leader":"true",           "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10063_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         
"core_node200":{           "core":"testNodeLost_shard20_replica_n200",          
 "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,     
      "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10006_solr",      
     "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node199":{           
"core":"testNodeLost_shard20_replica_n199",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10034_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node198":{           
"core":"testNodeLost_shard20_replica_n198",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10087_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node197":{           
"core":"testNodeLost_shard20_replica_n197",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10061_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node196":{           
"core":"testNodeLost_shard20_replica_n196",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10004_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node195":{           
"core":"testNodeLost_shard20_replica_n195",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10003_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node194":{           
"core":"testNodeLost_shard20_replica_n194",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10037_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node193":{           
"core":"testNodeLost_shard20_replica_n193",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10056_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node192":{           
"core":"testNodeLost_shard20_replica_n192",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10065_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"73330000-7fffffff",       
"state":"active"},     "shard10":{       "replicas":{         "core_node92":{   
        "core":"testNodeLost_shard10_replica_n92",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10065_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node91":{           
"core":"testNodeLost_shard10_replica_n91",           "leader":"true",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10063_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node94":{           
"core":"testNodeLost_shard10_replica_n94",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10037_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node93":{           
"core":"testNodeLost_shard10_replica_n93",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10056_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node96":{           
"core":"testNodeLost_shard10_replica_n96",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10004_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node95":{           
"core":"testNodeLost_shard10_replica_n95",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10003_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node100":{           
"core":"testNodeLost_shard10_replica_n100",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10006_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node98":{           
"core":"testNodeLost_shard10_replica_n98",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10087_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node97":{           
"core":"testNodeLost_shard10_replica_n97",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10061_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node99":{           
"core":"testNodeLost_shard10_replica_n99",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10034_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"f3330000-ffffffff",       
"state":"active"},     "shard11":{       "replicas":{         "core_node108":{  
         "core":"testNodeLost_shard11_replica_n108",           "leader":"true", 
          "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10031_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         
"core_node107":{           "core":"testNodeLost_shard11_replica_n107",          
 "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,     
      "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10044_solr",      
     "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node106":{           
"core":"testNodeLost_shard11_replica_n106",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10074_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node105":{           
"core":"testNodeLost_shard11_replica_n105",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10035_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node104":{           
"core":"testNodeLost_shard11_replica_n104",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10024_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node103":{           
"core":"testNodeLost_shard11_replica_n103",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10021_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node102":{           
"core":"testNodeLost_shard11_replica_n102",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10095_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node101":{           
"core":"testNodeLost_shard11_replica_n101",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10040_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node110":{           
"core":"testNodeLost_shard11_replica_n110",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10001_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node109":{           
"core":"testNodeLost_shard11_replica_n109",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10047_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"0-ccbffff",       
"state":"active"},     "shard12":{       "replicas":{         "core_node119":{  
         "core":"testNodeLost_shard12_replica_n119",           "leader":"true", 
          "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10060_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         
"core_node118":{           "core":"testNodeLost_shard12_replica_n118",          
 "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,     
      "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10018_solr",      
     "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node117":{           
"core":"testNodeLost_shard12_replica_n117",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10005_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node116":{           
"core":"testNodeLost_shard12_replica_n116",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10027_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node115":{           
"core":"testNodeLost_shard12_replica_n115",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10039_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node114":{           
"core":"testNodeLost_shard12_replica_n114",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10008_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node113":{           
"core":"testNodeLost_shard12_replica_n113",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10023_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node112":{           
"core":"testNodeLost_shard12_replica_n112",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10032_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node111":{           
"core":"testNodeLost_shard12_replica_n111",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10049_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node120":{           
"core":"testNodeLost_shard12_replica_n120",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10099_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"ccc0000-1998ffff",       
"state":"active"},     "shard13":{       "replicas":{         "core_node129":{  
         "core":"testNodeLost_shard13_replica_n129",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10084_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node128":{           
"core":"testNodeLost_shard13_replica_n128",           "leader":"true",          
 "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,     
      "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10042_solr",      
     "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node127":{           
"core":"testNodeLost_shard13_replica_n127",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10073_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node126":{           
"core":"testNodeLost_shard13_replica_n126",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10079_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node125":{           
"core":"testNodeLost_shard13_replica_n125",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10050_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node124":{           
"core":"testNodeLost_shard13_replica_n124",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10089_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node123":{           
"core":"testNodeLost_shard13_replica_n123",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10053_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node122":{           
"core":"testNodeLost_shard13_replica_n122",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10067_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node121":{           
"core":"testNodeLost_shard13_replica_n121",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10046_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node130":{           
"core":"testNodeLost_shard13_replica_n130",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10015_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"19990000-2665ffff",       
"state":"active"},     "shard2":{       "replicas":{         "core_node18":{    
       "core":"testNodeLost_shard2_replica_n18",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10018_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node17":{           
"core":"testNodeLost_shard2_replica_n17",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10005_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node19":{           
"core":"testNodeLost_shard2_replica_n19",           "leader":"true",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10060_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node20":{           
"core":"testNodeLost_shard2_replica_n20",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10099_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node12":{           
"core":"testNodeLost_shard2_replica_n12",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10032_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node11":{           
"core":"testNodeLost_shard2_replica_n11",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10049_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node14":{           
"core":"testNodeLost_shard2_replica_n14",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10008_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node13":{           
"core":"testNodeLost_shard2_replica_n13",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10023_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node16":{           
"core":"testNodeLost_shard2_replica_n16",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10027_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node15":{           
"core":"testNodeLost_shard2_replica_n15",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10039_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"8ccc0000-9998ffff",       
"state":"active"},     "shard14":{       "replicas":{         "core_node139":{  
         "core":"testNodeLost_shard14_replica_n139",           "leader":"true", 
          "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10076_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         
"core_node138":{           "core":"testNodeLost_shard14_replica_n138",          
 "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,     
      "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10030_solr",      
     "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node137":{           
"core":"testNodeLost_shard14_replica_n137",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10012_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node136":{           
"core":"testNodeLost_shard14_replica_n136",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10071_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node135":{           
"core":"testNodeLost_shard14_replica_n135",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10091_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node134":{           
"core":"testNodeLost_shard14_replica_n134",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10052_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node133":{           
"core":"testNodeLost_shard14_replica_n133",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10014_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node132":{           
"core":"testNodeLost_shard14_replica_n132",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10078_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node131":{           
"core":"testNodeLost_shard14_replica_n131",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10041_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node140":{           
"core":"testNodeLost_shard14_replica_n140",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10025_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"26660000-3332ffff",       
"state":"active"},     "shard3":{       "replicas":{         "core_node29":{    
       "core":"testNodeLost_shard3_replica_n29",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10084_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node28":{           
"core":"testNodeLost_shard3_replica_n28",           "leader":"true",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10042_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node30":{           
"core":"testNodeLost_shard3_replica_n30",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10015_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node21":{           
"core":"testNodeLost_shard3_replica_n21",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10046_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node23":{           
"core":"testNodeLost_shard3_replica_n23",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10053_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node22":{           
"core":"testNodeLost_shard3_replica_n22",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10067_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node25":{           
"core":"testNodeLost_shard3_replica_n25",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10050_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node24":{           
"core":"testNodeLost_shard3_replica_n24",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10089_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node27":{           
"core":"testNodeLost_shard3_replica_n27",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10073_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node26":{           
"core":"testNodeLost_shard3_replica_n26",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10079_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"99990000-a665ffff",       
"state":"active"},     "shard4":{       "replicas":{         "core_node39":{    
       "core":"testNodeLost_shard4_replica_n39",           "leader":"true",     
      "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10076_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node40":{ 
          "core":"testNodeLost_shard4_replica_n40",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10025_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node32":{           
"core":"testNodeLost_shard4_replica_n32",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10078_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node31":{           
"core":"testNodeLost_shard4_replica_n31",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10041_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node34":{           
"core":"testNodeLost_shard4_replica_n34",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10052_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node33":{           
"core":"testNodeLost_shard4_replica_n33",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10014_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node36":{           
"core":"testNodeLost_shard4_replica_n36",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10071_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node35":{           
"core":"testNodeLost_shard4_replica_n35",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10091_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node38":{           
"core":"testNodeLost_shard4_replica_n38",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10030_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node37":{           
"core":"testNodeLost_shard4_replica_n37",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10012_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"a6660000-b332ffff",       
"state":"active"},     "shard15":{       "replicas":{         "core_node149":{  
         "core":"testNodeLost_shard15_replica_n149",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10051_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node148":{           
"core":"testNodeLost_shard15_replica_n148",           "leader":"true",          
 "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,     
      "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10055_solr",      
     "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node147":{           
"core":"testNodeLost_shard15_replica_n147",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10064_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node146":{           
"core":"testNodeLost_shard15_replica_n146",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10036_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node145":{           
"core":"testNodeLost_shard15_replica_n145",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10086_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node144":{           
"core":"testNodeLost_shard15_replica_n144",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10029_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node143":{           
"core":"testNodeLost_shard15_replica_n143",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10011_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node142":{           
"core":"testNodeLost_shard15_replica_n142",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10022_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node141":{           
"core":"testNodeLost_shard15_replica_n141",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10088_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node150":{           
"core":"testNodeLost_shard15_replica_n150",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10026_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"33330000-3fffffff",       
"state":"active"},     "shard16":{       "replicas":{         "core_node159":{  
         "core":"testNodeLost_shard16_replica_n159",           "leader":"true", 
          "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10097_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         
"core_node158":{           "core":"testNodeLost_shard16_replica_n158",          
 "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,     
      "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10016_solr",      
     "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node202":{           
"node_name":"127.0.0.1:10072_solr",           
"core":"testNodeLost_shard16_replica_n201",           "state":"active",         
  "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node157":{      
     "core":"testNodeLost_shard16_replica_n157",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10013_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node156":{           
"core":"testNodeLost_shard16_replica_n156",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10068_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node155":{           
"core":"testNodeLost_shard16_replica_n155",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10028_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node154":{           
"core":"testNodeLost_shard16_replica_n154",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10085_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node153":{           
"core":"testNodeLost_shard16_replica_n153",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10059_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node151":{           
"core":"testNodeLost_shard16_replica_n151",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10033_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node160":{           
"core":"testNodeLost_shard16_replica_n160",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10002_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"40000000-4ccbffff",       
"state":"active"},     "shard5":{       "replicas":{         "core_node50":{    
       "core":"testNodeLost_shard5_replica_n50",           "leader":"true",     
      "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10026_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node41":{ 
          "core":"testNodeLost_shard5_replica_n41",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10088_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node43":{           
"core":"testNodeLost_shard5_replica_n43",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10011_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node42":{           
"core":"testNodeLost_shard5_replica_n42",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10022_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node45":{           
"core":"testNodeLost_shard5_replica_n45",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10086_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node44":{           
"core":"testNodeLost_shard5_replica_n44",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10029_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node47":{           
"core":"testNodeLost_shard5_replica_n47",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10064_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node46":{           
"core":"testNodeLost_shard5_replica_n46",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10036_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node49":{           
"core":"testNodeLost_shard5_replica_n49",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10051_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node48":{           
"core":"testNodeLost_shard5_replica_n48",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10055_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"b3330000-bfffffff",       
"state":"active"},     "shard17":{       "replicas":{         "core_node169":{  
         "core":"testNodeLost_shard17_replica_n169",           "leader":"true", 
          "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10070_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         
"core_node168":{           "core":"testNodeLost_shard17_replica_n168",          
 "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,     
      "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10009_solr",      
     "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node167":{           
"core":"testNodeLost_shard17_replica_n167",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10058_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node166":{           
"core":"testNodeLost_shard17_replica_n166",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10092_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node165":{           
"core":"testNodeLost_shard17_replica_n165",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10069_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node164":{           
"core":"testNodeLost_shard17_replica_n164",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10098_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node163":{           
"core":"testNodeLost_shard17_replica_n163",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10093_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node162":{           
"core":"testNodeLost_shard17_replica_n162",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10007_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node161":{           
"core":"testNodeLost_shard17_replica_n161",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10019_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node170":{           
"core":"testNodeLost_shard17_replica_n170",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10048_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"4ccc0000-5998ffff",       
"state":"active"},     "shard18":{       "replicas":{         "core_node180":{  
         "core":"testNodeLost_shard18_replica_n180",           "leader":"true", 
          "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10090_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         
"core_node179":{           "core":"testNodeLost_shard18_replica_n179",          
 "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,     
      "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10066_solr",      
     "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node178":{           
"core":"testNodeLost_shard18_replica_n178",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10043_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node177":{           
"core":"testNodeLost_shard18_replica_n177",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10017_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node176":{           
"core":"testNodeLost_shard18_replica_n176",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10081_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node175":{           
"core":"testNodeLost_shard18_replica_n175",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10020_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node174":{           
"core":"testNodeLost_shard18_replica_n174",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10038_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node173":{           
"core":"testNodeLost_shard18_replica_n173",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10045_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node172":{           
"core":"testNodeLost_shard18_replica_n172",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10000_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node171":{           
"core":"testNodeLost_shard18_replica_n171",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10077_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"59990000-6665ffff",       
"state":"active"},     "shard19":{       "replicas":{         "core_node190":{  
         "core":"testNodeLost_shard19_replica_n190",           "leader":"true", 
          "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10010_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         
"core_node189":{           "core":"testNodeLost_shard19_replica_n189",          
 "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,     
      "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10096_solr",      
     "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node188":{           
"core":"testNodeLost_shard19_replica_n188",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10075_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node187":{           
"core":"testNodeLost_shard19_replica_n187",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10072_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node186":{           
"core":"testNodeLost_shard19_replica_n186",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10057_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node185":{           
"core":"testNodeLost_shard19_replica_n185",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10062_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node184":{           
"core":"testNodeLost_shard19_replica_n184",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10054_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node183":{           
"core":"testNodeLost_shard19_replica_n183",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10094_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node182":{           
"core":"testNodeLost_shard19_replica_n182",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10083_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node181":{           
"core":"testNodeLost_shard19_replica_n181",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10080_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"66660000-7332ffff",       
"state":"active"},     "shard1":{       "replicas":{         "core_node1":{     
      "core":"testNodeLost_shard1_replica_n1",           "leader":"true",       
    "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,  
         "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10040_solr",   
        "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node10":{           
"core":"testNodeLost_shard1_replica_n10",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10001_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node6":{           
"core":"testNodeLost_shard1_replica_n6",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10074_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node7":{           
"core":"testNodeLost_shard1_replica_n7",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10044_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node8":{           
"core":"testNodeLost_shard1_replica_n8",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10031_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node9":{           
"core":"testNodeLost_shard1_replica_n9",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10047_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node2":{           
"core":"testNodeLost_shard1_replica_n2",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10095_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node3":{           
"core":"testNodeLost_shard1_replica_n3",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10021_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node4":{           
"core":"testNodeLost_shard1_replica_n4",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10024_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node5":{           
"core":"testNodeLost_shard1_replica_n5",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10035_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"80000000-8ccbffff",       
"state":"active"},     "shard6":{       "replicas":{         "core_node204":{   
        "node_name":"127.0.0.1:10056_solr",           
"core":"testNodeLost_shard6_replica_n203",           "state":"active",          
 "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node60":{        
   "core":"testNodeLost_shard6_replica_n60",           "leader":"true",         
  "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,    
       "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10002_solr",     
      "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node51":{           
"core":"testNodeLost_shard6_replica_n51",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10033_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node54":{           
"core":"testNodeLost_shard6_replica_n54",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10085_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node53":{           
"core":"testNodeLost_shard6_replica_n53",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10059_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node56":{           
"core":"testNodeLost_shard6_replica_n56",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10068_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node55":{           
"core":"testNodeLost_shard6_replica_n55",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10028_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node58":{           
"core":"testNodeLost_shard6_replica_n58",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10016_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node57":{           
"core":"testNodeLost_shard6_replica_n57",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10013_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node59":{           
"core":"testNodeLost_shard6_replica_n59",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10097_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"c0000000-cccbffff",       
"state":"active"},     "shard7":{       "replicas":{         "core_node70":{    
       "core":"testNodeLost_shard7_replica_n70",           "leader":"true",     
      "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10048_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node61":{ 
          "core":"testNodeLost_shard7_replica_n61",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10019_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node63":{           
"core":"testNodeLost_shard7_replica_n63",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10093_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node62":{           
"core":"testNodeLost_shard7_replica_n62",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10007_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node65":{           
"core":"testNodeLost_shard7_replica_n65",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10069_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node64":{           
"core":"testNodeLost_shard7_replica_n64",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10098_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node67":{           
"core":"testNodeLost_shard7_replica_n67",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10058_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node66":{           
"core":"testNodeLost_shard7_replica_n66",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10092_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node69":{           
"core":"testNodeLost_shard7_replica_n69",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10070_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node68":{           
"core":"testNodeLost_shard7_replica_n68",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10009_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"cccc0000-d998ffff",       
"state":"active"},     "shard8":{       "replicas":{         "core_node80":{    
       "core":"testNodeLost_shard8_replica_n80",           "leader":"true",     
      "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10090_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node72":{ 
          "core":"testNodeLost_shard8_replica_n72",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10000_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node71":{           "core":"testN

[...truncated too long message...]

 PropertiesRequestHandlerTest, TestAuthorizationFramework, TestStressReorder, 
PeerSyncWithIndexFingerprintCachingTest, LeaderElectionTest, 
TestLegacyNumericRangeQueryBuilder, TriggerSetPropertiesIntegrationTest, 
SpatialRPTFieldTypeTest, TestConfigSetImmutable, TestSuggestSpellingConverter, 
TestGraphTermsQParserPlugin, PreAnalyzedUpdateProcessorTest, 
DirectSolrSpellCheckerTest, TestAnalyzedSuggestions, TestNumericRangeQuery64, 
AnalyticsMergeStrategyTest, TestMultiWordSynonyms, TestXmlQParserPlugin, 
TestCollapseQParserPlugin, ShowFileRequestHandlerTest, TestJmxIntegration, 
TestJettySolrRunner, SpellCheckComponentTest, 
ConcurrentDeleteAndCreateCollectionTest, FileBasedSpellCheckerTest, 
MultiSolrCloudTestCaseTest, TestCorePropertiesReload, FullHLLTest, 
QueryResultKeyTest, CoreAdminCreateDiscoverTest, TestQueryTypes, 
DocValuesMissingTest, ClusterStateUpdateTest, UnloadDistributedZkTest, 
ShardRoutingTest, BasicZkTest, FullSolrCloudDistribCmdsTest, ZkSolrClientTest, 
TestZkChroot, TestRandomDVFaceting, ZkCLITest, ShardRoutingCustomTest, 
TestDistributedSearch, TestRecovery, TestHashPartitioner, 
DistributedSpellCheckComponentTest, ZkControllerTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ZkControllerTest 
-Dtests.seed=7B3F86A0E05CE243 -Dtests.multiplier=2 -Dtests.slow=true 
-Dtests.badapples=true -Dtests.locale=es-PE -Dtests.timezone=Etc/Greenwich 
-Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J2 | ZkControllerTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ObjectTracker found 1 
object(s) that were not released!!! [Overseer]
   [junit4]    > 
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: 
org.apache.solr.cloud.Overseer
   [junit4]    >        at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
   [junit4]    >        at 
org.apache.solr.cloud.Overseer.start(Overseer.java:545)
   [junit4]    >        at 
org.apache.solr.cloud.OverseerElectionContext.runLeaderProcess(ElectionContext.java:851)
   [junit4]    >        at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
   [junit4]    >        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
   [junit4]    >        at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)
   [junit4]    >        at 
org.apache.solr.cloud.LeaderElector.retryElection(LeaderElector.java:393)
   [junit4]    >        at 
org.apache.solr.cloud.ZkController.rejoinOverseerElection(ZkController.java:2081)
   [junit4]    >        at 
org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:331)
   [junit4]    >        at java.lang.Thread.run(Thread.java:748)
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([7B3F86A0E05CE243]:0)
   [junit4]    >        at 
org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:303)
   [junit4]    >        at java.lang.Thread.run(Thread.java:748)Throwable #2: 
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE 
scope at org.apache.solr.cloud.ZkControllerTest: 
   [junit4]    >    1) Thread[id=21183, 
name=OverseerAutoScalingTriggerThread-73773449468116995-127.0.0.1:8983_solr-n_0000000001,
 state=RUNNABLE, group=Overseer autoscaling triggers]
   [junit4]    >         at 
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:329)
   [junit4]    >         at 
java.lang.ClassLoader.loadClass(ClassLoader.java:357)
   [junit4]    >         at 
org.apache.logging.log4j.core.impl.ThrowableProxy.loadClass(ThrowableProxy.java:613)
   [junit4]    >         at 
org.apache.logging.log4j.core.impl.ThrowableProxy.toExtendedStackTrace(ThrowableProxy.java:738)
   [junit4]    >         at 
org.apache.logging.log4j.core.impl.ThrowableProxy.<init>(ThrowableProxy.java:138)
   [junit4]    >         at 
org.apache.logging.log4j.core.impl.ThrowableProxy.<init>(ThrowableProxy.java:122)
   [junit4]    >         at 
org.apache.logging.log4j.core.impl.Log4jLogEvent.getThrownProxy(Log4jLogEvent.java:605)
   [junit4]    >         at 
org.apache.logging.log4j.core.pattern.ExtendedThrowablePatternConverter.format(ExtendedThrowablePatternConverter.java:64)
   [junit4]    >         at 
org.apache.logging.log4j.core.pattern.PatternFormatter.format(PatternFormatter.java:38)
   [junit4]    >         at 
org.apache.logging.log4j.core.layout.PatternLayout$PatternSerializer.toSerializable(PatternLayout.java:334)
   [junit4]    >         at 
org.apache.logging.log4j.core.layout.PatternLayout.toText(PatternLayout.java:233)
   [junit4]    >         at 
org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:218)
   [junit4]    >         at 
org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:58)
   [junit4]    >         at 
org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:177)
   [junit4]    >         at 
org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:170)
   [junit4]    >         at 
org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:161)
   [junit4]    >         at 
org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156)
   [junit4]    >         at 
org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129)
   [junit4]    >         at 
org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120)
   [junit4]    >         at 
org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
   [junit4]    >         at 
org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:448)
   [junit4]    >         at 
org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:433)
   [junit4]    >         at 
org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:417)
   [junit4]    >         at 
org.apache.logging.log4j.core.config.LoggerConfig.logParent(LoggerConfig.java:439)
   [junit4]    >         at 
org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:434)
   [junit4]    >         at 
org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:417)
   [junit4]    >         at 
org.apache.logging.log4j.core.config.LoggerConfig.logParent(LoggerConfig.java:439)
   [junit4]    >         at 
org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:434)
   [junit4]    >         at 
org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:417)
   [junit4]    >         at 
org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:403)
   [junit4]    >         at 
org.apache.logging.log4j.core.config.DefaultReliabilityStrategy.log(DefaultReliabilityStrategy.java:49)
   [junit4]    >         at 
org.apache.logging.log4j.core.Logger.logMessage(Logger.java:146)
   [junit4]    >         at 
org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2170)
   [junit4]    >         at 
org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2125)
   [junit4]    >         at 
org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2108)
   [junit4]    >         at 
org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2002)
   [junit4]    >         at 
org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1974)
   [junit4]    >         at 
org.apache.logging.slf4j.Log4jLogger.error(Log4jLogger.java:319)
   [junit4]    >         at 
org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:146)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([7B3F86A0E05CE243]:0)
   [junit4] Completed [386/806 (4!)] on J2 in 167.52s, 4 tests, 1 failure, 1 
error <<< FAILURES!

[...truncated 51543 lines...]
[asciidoctor:convert] asciidoctor: ERROR: about-this-guide.adoc: line 1: 
invalid part, must have at least one section (e.g., chapter, appendix, etc.)
[asciidoctor:convert] asciidoctor: ERROR: solr-glossary.adoc: line 1: invalid 
part, must have at least one section (e.g., chapter, appendix, etc.)
     [java] ID occurs multiple times: moving-average
     [java]  ... 
file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-ref-guide/bare-bones-html/time-series.html
     [java]  ... 
file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-ref-guide/bare-bones-html/dsp.html
     [java] Relative link points at id that doesn't exist in dest: 
dsp.html#digital-signal-processing
     [java]  ... source: 
file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-ref-guide/bare-bones-html/math-expressions.html
     [java] Processed 2207 links (1763 relative) to 3079 anchors in 245 files
     [java] Total of 2 problems found

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/build.xml:642:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/build.xml:101:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build.xml:670:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build.xml:192:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/solr-ref-guide/build.xml:326:
 Java returned: 255

Total time: 175 minutes 18 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to