Build: https://ci-builds.apache.org/job/Solr/job/Solr-NightlyTests-main/1274/

11 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test

Error Message:
java.lang.AssertionError: There are still nodes recovering - waited for 
600SECONDS

Stack Trace:
java.lang.AssertionError: There are still nodes recovering - waited for 
600SECONDS
        at 
__randomizedtesting.SeedInfo.seed([ADD8807A4E373E10:258CBFA0E0CB53E8]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:225)
        at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:158)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:1171)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1886)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1878)
        at 
org.apache.solr.cloud.AbstractChaosMonkeyNothingIsSafeTestBase.test(AbstractChaosMonkeyNothingIsSafeTestBase.java:240)
        at 
java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1069)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1040)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
        at 
org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at 
org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at 
org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at 
org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
        at java.base/java.lang.Thread.run(Thread.java:1583)


FAILED:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test

Error Message:
java.lang.AssertionError: There are still nodes recovering - waited for 
600SECONDS

Stack Trace:
java.lang.AssertionError: There are still nodes recovering - waited for 
600SECONDS
        at 
__randomizedtesting.SeedInfo.seed([ADD8807A4E373E10:258CBFA0E0CB53E8]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:225)
        at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:158)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:1171)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1886)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1878)
        at 
org.apache.solr.cloud.AbstractChaosMonkeyNothingIsSafeTestBase.test(AbstractChaosMonkeyNothingIsSafeTestBase.java:240)
        at 
java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1069)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1040)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
        at 
org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at 
org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at 
org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at 
org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
        at java.base/java.lang.Thread.run(Thread.java:1583)


FAILED:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test

Error Message:
java.lang.AssertionError: There are still nodes recovering - waited for 
600SECONDS

Stack Trace:
java.lang.AssertionError: There are still nodes recovering - waited for 
600SECONDS
        at 
__randomizedtesting.SeedInfo.seed([ADD8807A4E373E10:258CBFA0E0CB53E8]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:225)
        at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:158)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:1171)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1886)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1878)
        at 
org.apache.solr.cloud.AbstractChaosMonkeyNothingIsSafeTestBase.test(AbstractChaosMonkeyNothingIsSafeTestBase.java:240)
        at 
java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1069)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1040)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
        at 
org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at 
org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at 
org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at 
org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
        at java.base/java.lang.Thread.run(Thread.java:1583)


FAILED:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test

Error Message:
org.apache.solr.common.SolrException: No registered leader was found after 
waiting for 30000ms , collection: collection1 slice: shard2 saw 
state=DocCollection(collection1//collections/collection1/state.json/20 )={
  "pullReplicas":"0",
  "configName":"conf1",
  "nrtReplicas":1,
  "replicationFactor":1,
  "tlogReplicas":"0",
  "router":{"name":"compositeId"},
  "shards":{
    "shard1":{
      "range":"80000000-d554ffff",
      "replicas":{
        "core_node5":{
          "core":"collection1_shard1_replica_n2",
          "node_name":"127.0.0.1:46851_solr",
          "type":"NRT",
          "state":"active",
          "leader":"true",
          "force_set_state":"false",
          "base_url":"http://127.0.0.1:46851/solr"},
        "core_node11":{
          "core":"collection1_shard1_replica_n8",
          "node_name":"127.0.0.1:46655_solr",
          "type":"NRT",
          "state":"active",
          "force_set_state":"false",
          "base_url":"http://127.0.0.1:46655/solr"},
        "core_node17":{
          "core":"collection1_shard1_replica_n14",
          "node_name":"127.0.0.1:34027_solr",
          "type":"NRT",
          "state":"active",
          "force_set_state":"false",
          "base_url":"http://127.0.0.1:34027/solr"},
        "core_node23":{
          "core":"collection1_shard1_replica_n20",
          "node_name":"127.0.0.1:40835_solr",
          "type":"NRT",
          "state":"active",
          "force_set_state":"false",
          "base_url":"http://127.0.0.1:40835/solr"}},
      "state":"active"},
    "shard2":{
      "range":"d5550000-2aa9ffff",
      "replicas":{
        "core_node6":{
          "core":"collection1_shard2_replica_n3",
          "node_name":"127.0.0.1:44841_solr",
          "type":"NRT",
          "state":"down",
          "force_set_state":"false",
          "base_url":"http://127.0.0.1:44841/solr"},
        "core_node10":{
          "core":"collection1_shard2_replica_n7",
          "node_name":"127.0.0.1:38645_solr",
          "type":"NRT",
          "state":"down",
          "force_set_state":"false",
          "base_url":"http://127.0.0.1:38645/solr"},
        "core_node16":{
          "core":"collection1_shard2_replica_n13",
          "node_name":"127.0.0.1:41375_solr",
          "type":"NRT",
          "state":"active",
          "force_set_state":"false",
          "base_url":"http://127.0.0.1:41375/solr"},
        "core_node22":{
          "core":"collection1_shard2_replica_n19",
          "node_name":"127.0.0.1:39501_solr",
          "type":"NRT",
          "state":"active",
          "force_set_state":"false",
          "base_url":"http://127.0.0.1:39501/solr"}},
      "state":"active"},
    "shard3":{
      "range":"2aaa0000-7fffffff",
      "replicas":{
        "core_node4":{
          "core":"collection1_shard3_replica_n1",
          "node_name":"127.0.0.1:41547_solr",
          "type":"NRT",
          "state":"active",
          "leader":"true",
          "force_set_state":"false",
          "base_url":"http://127.0.0.1:41547/solr"},
        "core_node12":{
          "core":"collection1_shard3_replica_n9",
          "node_name":"127.0.0.1:35281_solr",
          "type":"NRT",
          "state":"active",
          "force_set_state":"false",
          "base_url":"http://127.0.0.1:35281/solr"},
        "core_node18":{
          "core":"collection1_shard3_replica_n15",
          "node_name":"127.0.0.1:34227_solr",
          "type":"NRT",
          "state":"active",
          "force_set_state":"false",
          "base_url":"http://127.0.0.1:34227/solr"},
        "core_node24":{
          "core":"collection1_shard3_replica_n21",
          "node_name":"127.0.0.1:38177_solr",
          "type":"NRT",
          "state":"active",
          "force_set_state":"false",
          "base_url":"http://127.0.0.1:38177/solr"}},
      "state":"active"}}} with live_nodes=[127.0.0.1:46851_solr, 
127.0.0.1:39501_solr, 127.0.0.1:41375_solr, 127.0.0.1:41547_solr, 
127.0.0.1:34027_solr, 127.0.0.1:34227_solr, 127.0.0.1:46655_solr, 
127.0.0.1:40835_solr, 127.0.0.1:38645_solr, 127.0.0.1:35281_solr, 
127.0.0.1:38177_solr, 127.0.0.1:40145_solr]

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after 
waiting for 30000ms , collection: collection1 slice: shard2 saw 
state=DocCollection(collection1//collections/collection1/state.json/20 )={
  "pullReplicas":"0",
  "configName":"conf1",
  "nrtReplicas":1,
  "replicationFactor":1,
  "tlogReplicas":"0",
  "router":{"name":"compositeId"},
  "shards":{
    "shard1":{
      "range":"80000000-d554ffff",
      "replicas":{
        "core_node5":{
          "core":"collection1_shard1_replica_n2",
          "node_name":"127.0.0.1:46851_solr",
          "type":"NRT",
          "state":"active",
          "leader":"true",
          "force_set_state":"false",
          "base_url":"http://127.0.0.1:46851/solr"},
        "core_node11":{
          "core":"collection1_shard1_replica_n8",
          "node_name":"127.0.0.1:46655_solr",
          "type":"NRT",
          "state":"active",
          "force_set_state":"false",
          "base_url":"http://127.0.0.1:46655/solr"},
        "core_node17":{
          "core":"collection1_shard1_replica_n14",
          "node_name":"127.0.0.1:34027_solr",
          "type":"NRT",
          "state":"active",
          "force_set_state":"false",
          "base_url":"http://127.0.0.1:34027/solr"},
        "core_node23":{
          "core":"collection1_shard1_replica_n20",
          "node_name":"127.0.0.1:40835_solr",
          "type":"NRT",
          "state":"active",
          "force_set_state":"false",
          "base_url":"http://127.0.0.1:40835/solr"}},
      "state":"active"},
    "shard2":{
      "range":"d5550000-2aa9ffff",
      "replicas":{
        "core_node6":{
          "core":"collection1_shard2_replica_n3",
          "node_name":"127.0.0.1:44841_solr",
          "type":"NRT",
          "state":"down",
          "force_set_state":"false",
          "base_url":"http://127.0.0.1:44841/solr"},
        "core_node10":{
          "core":"collection1_shard2_replica_n7",
          "node_name":"127.0.0.1:38645_solr",
          "type":"NRT",
          "state":"down",
          "force_set_state":"false",
          "base_url":"http://127.0.0.1:38645/solr"},
        "core_node16":{
          "core":"collection1_shard2_replica_n13",
          "node_name":"127.0.0.1:41375_solr",
          "type":"NRT",
          "state":"active",
          "force_set_state":"false",
          "base_url":"http://127.0.0.1:41375/solr"},
        "core_node22":{
          "core":"collection1_shard2_replica_n19",
          "node_name":"127.0.0.1:39501_solr",
          "type":"NRT",
          "state":"active",
          "force_set_state":"false",
          "base_url":"http://127.0.0.1:39501/solr"}},
      "state":"active"},
    "shard3":{
      "range":"2aaa0000-7fffffff",
      "replicas":{
        "core_node4":{
          "core":"collection1_shard3_replica_n1",
          "node_name":"127.0.0.1:41547_solr",
          "type":"NRT",
          "state":"active",
          "leader":"true",
          "force_set_state":"false",
          "base_url":"http://127.0.0.1:41547/solr"},
        "core_node12":{
          "core":"collection1_shard3_replica_n9",
          "node_name":"127.0.0.1:35281_solr",
          "type":"NRT",
          "state":"active",
          "force_set_state":"false",
          "base_url":"http://127.0.0.1:35281/solr"},
        "core_node18":{
          "core":"collection1_shard3_replica_n15",
          "node_name":"127.0.0.1:34227_solr",
          "type":"NRT",
          "state":"active",
          "force_set_state":"false",
          "base_url":"http://127.0.0.1:34227/solr"},
        "core_node24":{
          "core":"collection1_shard3_replica_n21",
          "node_name":"127.0.0.1:38177_solr",
          "type":"NRT",
          "state":"active",
          "force_set_state":"false",
          "base_url":"http://127.0.0.1:38177/solr"}},
      "state":"active"}}} with live_nodes=[127.0.0.1:46851_solr, 
127.0.0.1:39501_solr, 127.0.0.1:41375_solr, 127.0.0.1:41547_solr, 
127.0.0.1:34027_solr, 127.0.0.1:34227_solr, 127.0.0.1:46655_solr, 
127.0.0.1:40835_solr, 127.0.0.1:38645_solr, 127.0.0.1:35281_solr, 
127.0.0.1:38177_solr, 127.0.0.1:40145_solr]
        at 
__randomizedtesting.SeedInfo.seed([ADD8807A4E373E10:258CBFA0E0CB53E8]:0)
        at 
app//org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:978)
        at 
app//org.apache.solr.cloud.AbstractChaosMonkeyNothingIsSafeTestBase.test(AbstractChaosMonkeyNothingIsSafeTestBase.java:244)
        at 
[email protected]/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at [email protected]/java.lang.reflect.Method.invoke(Method.java:580)
        at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
        at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
        at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
        at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
        at 
app//org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1069)
        at 
app//org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1040)
        at 
app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
        at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
        at 
app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
        at 
app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at 
app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at 
app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
        at 
app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
        at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
        at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
        at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
        at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
        at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
        at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
        at 
app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at 
app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at 
app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
        at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at 
app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
        at [email protected]/java.lang.Thread.run(Thread.java:1583)


FAILED:  org.apache.solr.hdfs.cloud.HdfsChaosMonkeyNothingIsSafeTest.test

Error Message:
java.lang.AssertionError: There are still nodes recovering - waited for 
600SECONDS

Stack Trace:
java.lang.AssertionError: There are still nodes recovering - waited for 
600SECONDS
        at 
__randomizedtesting.SeedInfo.seed([ADD8807A4E373E10:258CBFA0E0CB53E8]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:225)
        at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:158)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:1171)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1886)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1878)
        at 
org.apache.solr.cloud.AbstractChaosMonkeyNothingIsSafeTestBase.test(AbstractChaosMonkeyNothingIsSafeTestBase.java:240)
        at 
java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1069)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1040)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
        at 
org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at 
org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at 
org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at 
org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
        at java.base/java.lang.Thread.run(Thread.java:1583)


FAILED:  org.apache.solr.hdfs.cloud.HdfsChaosMonkeyNothingIsSafeTest.test

Error Message:
java.lang.AssertionError: There are still nodes recovering - waited for 
600SECONDS

Stack Trace:
java.lang.AssertionError: There are still nodes recovering - waited for 
600SECONDS
        at 
__randomizedtesting.SeedInfo.seed([ADD8807A4E373E10:258CBFA0E0CB53E8]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:225)
        at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:158)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:1171)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1886)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1878)
        at 
org.apache.solr.cloud.AbstractChaosMonkeyNothingIsSafeTestBase.test(AbstractChaosMonkeyNothingIsSafeTestBase.java:240)
        at 
java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1069)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1040)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
        at 
org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at 
org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at 
org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at 
org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
        at java.base/java.lang.Thread.run(Thread.java:1583)


FAILED:  org.apache.solr.hdfs.cloud.HdfsChaosMonkeyNothingIsSafeTest.test

Error Message:
java.lang.AssertionError: There are still nodes recovering - waited for 
600SECONDS

Stack Trace:
java.lang.AssertionError: There are still nodes recovering - waited for 
600SECONDS
        at 
__randomizedtesting.SeedInfo.seed([ADD8807A4E373E10:258CBFA0E0CB53E8]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:225)
        at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:158)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:1171)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1886)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1878)
        at 
org.apache.solr.cloud.AbstractChaosMonkeyNothingIsSafeTestBase.test(AbstractChaosMonkeyNothingIsSafeTestBase.java:240)
        at 
java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1069)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1040)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
        at 
org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at 
org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at 
org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at 
org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
        at java.base/java.lang.Thread.run(Thread.java:1583)


FAILED:  org.apache.solr.hdfs.cloud.HdfsChaosMonkeyNothingIsSafeTest.test

Error Message:
org.apache.solr.common.SolrException: No registered leader was found after 
waiting for 30000ms , collection: collection1 slice: shard2 saw 
state=DocCollection(collection1//collections/collection1/state.json/29 )={
  "pullReplicas":"0",
  "configName":"conf1",
  "nrtReplicas":1,
  "replicationFactor":1,
  "tlogReplicas":"0",
  "router":{"name":"compositeId"},
  "shards":{
    "shard1":{
      "range":"80000000-d554ffff",
      "replicas":{
        "core_node2":{
          "core":"collection1_shard1_replica_n1",
          "node_name":"127.0.0.1:40191_solr",
          "type":"NRT",
          "state":"active",
          "leader":"true",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:40191/solr";,
          
"dataDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node2/data/",
          "shared_storage":"true",
          
"ulogDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node2/data/"},
        "core_node11":{
          "core":"collection1_shard1_replica_n8",
          "node_name":"127.0.0.1:43453_solr",
          "type":"NRT",
          "state":"active",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:43453/solr";,
          
"dataDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node11/data/",
          "shared_storage":"true",
          
"ulogDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node11/data/"},
        "core_node16":{
          "core":"collection1_shard1_replica_n13",
          "node_name":"127.0.0.1:35847_solr",
          "type":"NRT",
          "state":"active",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:35847/solr";,
          
"dataDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node16/data/",
          "shared_storage":"true",
          
"ulogDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node16/data/"},
        "core_node21":{
          "core":"collection1_shard1_replica_n19",
          "node_name":"127.0.0.1:38109_solr",
          "type":"NRT",
          "state":"active",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:38109/solr";,
          
"dataDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node21/data/",
          "shared_storage":"true",
          
"ulogDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node21/data/"}},
      "state":"active"},
    "shard2":{
      "range":"d5550000-2aa9ffff",
      "replicas":{
        "core_node4":{
          "core":"collection1_shard2_replica_n3",
          "node_name":"127.0.0.1:44011_solr",
          "type":"NRT",
          "state":"down",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:44011/solr";,
          
"dataDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node4/data/",
          "shared_storage":"true",
          
"ulogDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node4/data/"},
        "core_node10":{
          "core":"collection1_shard2_replica_n7",
          "node_name":"127.0.0.1:35467_solr",
          "type":"NRT",
          "state":"down",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:35467/solr";,
          
"dataDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node10/data/",
          "shared_storage":"true",
          
"ulogDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node10/data/"},
        "core_node18":{
          "core":"collection1_shard2_replica_n15",
          "node_name":"127.0.0.1:41251_solr",
          "type":"NRT",
          "state":"active",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:41251/solr";,
          
"dataDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node18/data/",
          "shared_storage":"true",
          
"ulogDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node18/data/"},
        "core_node24":{
          "core":"collection1_shard2_replica_n23",
          "node_name":"127.0.0.1:41993_solr",
          "type":"NRT",
          "state":"active",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:41993/solr";,
          
"dataDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node24/data/",
          "shared_storage":"true",
          
"ulogDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node24/data/"}},
      "state":"active"},
    "shard3":{
      "range":"2aaa0000-7fffffff",
      "replicas":{
        "core_node6":{
          "core":"collection1_shard3_replica_n5",
          "node_name":"127.0.0.1:46733_solr",
          "type":"NRT",
          "state":"active",
          "leader":"true",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:46733/solr";,
          
"dataDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node6/data/",
          "shared_storage":"true",
          
"ulogDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node6/data/"},
        "core_node12":{
          "core":"collection1_shard3_replica_n9",
          "node_name":"127.0.0.1:41489_solr",
          "type":"NRT",
          "state":"active",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:41489/solr";,
          
"dataDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node12/data/",
          "shared_storage":"true",
          
"ulogDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node12/data/"},
        "core_node17":{
          "core":"collection1_shard3_replica_n14",
          "node_name":"127.0.0.1:40795_solr",
          "type":"NRT",
          "state":"active",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:40795/solr";,
          
"dataDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node17/data/",
          "shared_storage":"true",
          
"ulogDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node17/data/"},
        "core_node22":{
          "core":"collection1_shard3_replica_n20",
          "node_name":"127.0.0.1:43211_solr",
          "type":"NRT",
          "state":"active",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:43211/solr";,
          
"dataDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node22/data/",
          "shared_storage":"true",
          
"ulogDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node22/data/"}},
      "state":"active"}}} with live_nodes=[127.0.0.1:46733_solr, 
127.0.0.1:38109_solr, 127.0.0.1:41251_solr, 127.0.0.1:43453_solr, 
127.0.0.1:42447_solr, 127.0.0.1:41489_solr, 127.0.0.1:35847_solr, 
127.0.0.1:41993_solr, 127.0.0.1:40191_solr, 127.0.0.1:43211_solr, 
127.0.0.1:35467_solr, 127.0.0.1:40795_solr]

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after 
waiting for 30000ms , collection: collection1 slice: shard2 saw 
state=DocCollection(collection1//collections/collection1/state.json/29 )={
  "pullReplicas":"0",
  "configName":"conf1",
  "nrtReplicas":1,
  "replicationFactor":1,
  "tlogReplicas":"0",
  "router":{"name":"compositeId"},
  "shards":{
    "shard1":{
      "range":"80000000-d554ffff",
      "replicas":{
        "core_node2":{
          "core":"collection1_shard1_replica_n1",
          "node_name":"127.0.0.1:40191_solr",
          "type":"NRT",
          "state":"active",
          "leader":"true",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:40191/solr";,
          
"dataDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node2/data/",
          "shared_storage":"true",
          
"ulogDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node2/data/"},
        "core_node11":{
          "core":"collection1_shard1_replica_n8",
          "node_name":"127.0.0.1:43453_solr",
          "type":"NRT",
          "state":"active",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:43453/solr";,
          
"dataDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node11/data/",
          "shared_storage":"true",
          
"ulogDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node11/data/"},
        "core_node16":{
          "core":"collection1_shard1_replica_n13",
          "node_name":"127.0.0.1:35847_solr",
          "type":"NRT",
          "state":"active",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:35847/solr";,
          
"dataDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node16/data/",
          "shared_storage":"true",
          
"ulogDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node16/data/"},
        "core_node21":{
          "core":"collection1_shard1_replica_n19",
          "node_name":"127.0.0.1:38109_solr",
          "type":"NRT",
          "state":"active",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:38109/solr";,
          
"dataDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node21/data/",
          "shared_storage":"true",
          
"ulogDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node21/data/"}},
      "state":"active"},
    "shard2":{
      "range":"d5550000-2aa9ffff",
      "replicas":{
        "core_node4":{
          "core":"collection1_shard2_replica_n3",
          "node_name":"127.0.0.1:44011_solr",
          "type":"NRT",
          "state":"down",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:44011/solr";,
          
"dataDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node4/data/",
          "shared_storage":"true",
          
"ulogDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node4/data/"},
        "core_node10":{
          "core":"collection1_shard2_replica_n7",
          "node_name":"127.0.0.1:35467_solr",
          "type":"NRT",
          "state":"down",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:35467/solr";,
          
"dataDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node10/data/",
          "shared_storage":"true",
          
"ulogDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node10/data/"},
        "core_node18":{
          "core":"collection1_shard2_replica_n15",
          "node_name":"127.0.0.1:41251_solr",
          "type":"NRT",
          "state":"active",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:41251/solr";,
          
"dataDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node18/data/",
          "shared_storage":"true",
          
"ulogDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node18/data/"},
        "core_node24":{
          "core":"collection1_shard2_replica_n23",
          "node_name":"127.0.0.1:41993_solr",
          "type":"NRT",
          "state":"active",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:41993/solr";,
          
"dataDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node24/data/",
          "shared_storage":"true",
          
"ulogDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node24/data/"}},
      "state":"active"},
    "shard3":{
      "range":"2aaa0000-7fffffff",
      "replicas":{
        "core_node6":{
          "core":"collection1_shard3_replica_n5",
          "node_name":"127.0.0.1:46733_solr",
          "type":"NRT",
          "state":"active",
          "leader":"true",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:46733/solr";,
          
"dataDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node6/data/",
          "shared_storage":"true",
          
"ulogDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node6/data/"},
        "core_node12":{
          "core":"collection1_shard3_replica_n9",
          "node_name":"127.0.0.1:41489_solr",
          "type":"NRT",
          "state":"active",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:41489/solr";,
          
"dataDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node12/data/",
          "shared_storage":"true",
          
"ulogDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node12/data/"},
        "core_node17":{
          "core":"collection1_shard3_replica_n14",
          "node_name":"127.0.0.1:40795_solr",
          "type":"NRT",
          "state":"active",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:40795/solr";,
          
"dataDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node17/data/",
          "shared_storage":"true",
          
"ulogDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node17/data/"},
        "core_node22":{
          "core":"collection1_shard3_replica_n20",
          "node_name":"127.0.0.1:43211_solr",
          "type":"NRT",
          "state":"active",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:43211/solr";,
          
"dataDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node22/data/",
          "shared_storage":"true",
          
"ulogDir":"hdfs://localhost.localdomain:37837/solr_hdfs_home/collection1/core_node22/data/"}},
      "state":"active"}}} with live_nodes=[127.0.0.1:46733_solr, 
127.0.0.1:38109_solr, 127.0.0.1:41251_solr, 127.0.0.1:43453_solr, 
127.0.0.1:42447_solr, 127.0.0.1:41489_solr, 127.0.0.1:35847_solr, 
127.0.0.1:41993_solr, 127.0.0.1:40191_solr, 127.0.0.1:43211_solr, 
127.0.0.1:35467_solr, 127.0.0.1:40795_solr]
        at 
__randomizedtesting.SeedInfo.seed([ADD8807A4E373E10:258CBFA0E0CB53E8]:0)
        at 
app//org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:978)
        at 
app//org.apache.solr.cloud.AbstractChaosMonkeyNothingIsSafeTestBase.test(AbstractChaosMonkeyNothingIsSafeTestBase.java:244)
        at 
[email protected]/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at [email protected]/java.lang.reflect.Method.invoke(Method.java:580)
        at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
        at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
        at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
        at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
        at 
app//org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1069)
        at 
app//org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1040)
        at 
app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
        at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
        at 
app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
        at 
app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at 
app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at 
app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
        at 
app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
        at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
        at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
        at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
        at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
        at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
        at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
        at 
app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at 
app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at 
app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
        at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at 
app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
        at [email protected]/java.lang.Thread.run(Thread.java:1583)


FAILED:  org.apache.solr.hdfs.cloud.HdfsChaosMonkeySafeLeaderTest.test

Error Message:
java.lang.AssertionError: expected:<0> but was:<5>

Stack Trace:
java.lang.AssertionError: expected:<0> but was:<5>
        at 
__randomizedtesting.SeedInfo.seed([ADD8807A4E373E10:258CBFA0E0CB53E8]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:633)
        at 
org.apache.solr.cloud.AbstractChaosMonkeySafeLeaderTestBase.test(AbstractChaosMonkeySafeLeaderTestBase.java:166)
        at 
java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1069)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1040)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
        at 
org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at 
org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at 
org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at 
org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
        at java.base/java.lang.Thread.run(Thread.java:1583)


FAILED:  org.apache.solr.hdfs.cloud.HdfsChaosMonkeySafeLeaderTest.test

Error Message:
java.lang.AssertionError: expected:<0> but was:<2>

Stack Trace:
java.lang.AssertionError: expected:<0> but was:<2>
        at 
__randomizedtesting.SeedInfo.seed([ADD8807A4E373E10:258CBFA0E0CB53E8]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:633)
        at 
org.apache.solr.cloud.AbstractChaosMonkeySafeLeaderTestBase.test(AbstractChaosMonkeySafeLeaderTestBase.java:166)
        at 
java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1069)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1040)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
        at 
org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at 
org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at 
org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at 
org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
        at java.base/java.lang.Thread.run(Thread.java:1583)


FAILED:  org.apache.solr.hdfs.cloud.HdfsRestartWhileUpdatingTest.test

Error Message:
java.lang.AssertionError: There are still nodes recovering - waited for 
320SECONDS

Stack Trace:
java.lang.AssertionError: There are still nodes recovering - waited for 
320SECONDS
        at 
__randomizedtesting.SeedInfo.seed([ADD8807A4E373E10:258CBFA0E0CB53E8]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:225)
        at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:158)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:1171)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1886)
        at 
org.apache.solr.cloud.AbstractRestartWhileUpdatingTestBase.test(AbstractRestartWhileUpdatingTestBase.java:143)
        at 
java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1069)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1040)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
        at 
org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at 
org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at 
org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at 
org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
        at java.base/java.lang.Thread.run(Thread.java:1583)




Build Log:
[...truncated 200046 lines...]
ERROR: The following test(s) have failed:
  - org.apache.solr.hdfs.cloud.HdfsChaosMonkeySafeLeaderTest.test 
(:solr:modules:hdfs)
    Test history: 
https://ge.apache.org/scans/tests?search.rootProjectNames=solr-root&tests.container=org.apache.solr.hdfs.cloud.HdfsChaosMonkeySafeLeaderTest&tests.test=test
 
http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.hdfs.cloud.HdfsChaosMonkeySafeLeaderTest.test
    Test output: 
/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/solr/modules/hdfs/build/test-results/test/outputs/OUTPUT-org.apache.solr.hdfs.cloud.HdfsChaosMonkeySafeLeaderTest.txt
    Reproduce with: ./gradlew :solr:modules:hdfs:test --tests 
"org.apache.solr.hdfs.cloud.HdfsChaosMonkeySafeLeaderTest.test" -Ptests.jvms=4 
-Ptests.haltonfailure=false "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 
-XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" 
-Ptests.seed=ADD8807A4E373E10 -Ptests.multiplier=2 -Ptests.nightly=true 
-Ptests.badapples=false -Ptests.file.encoding=UTF-8 
-Ptests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/test-data/enwiki.random.lines.txt

  - org.apache.solr.hdfs.cloud.HdfsRestartWhileUpdatingTest.test 
(:solr:modules:hdfs)
    Test history: 
https://ge.apache.org/scans/tests?search.rootProjectNames=solr-root&tests.container=org.apache.solr.hdfs.cloud.HdfsRestartWhileUpdatingTest&tests.test=test
 
http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.hdfs.cloud.HdfsRestartWhileUpdatingTest.test
    Test output: 
/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/solr/modules/hdfs/build/test-results/test/outputs/OUTPUT-org.apache.solr.hdfs.cloud.HdfsRestartWhileUpdatingTest.txt
    Reproduce with: ./gradlew :solr:modules:hdfs:test --tests 
"org.apache.solr.hdfs.cloud.HdfsRestartWhileUpdatingTest.test" -Ptests.jvms=4 
-Ptests.haltonfailure=false "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 
-XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" 
-Ptests.seed=ADD8807A4E373E10 -Ptests.multiplier=2 -Ptests.nightly=true 
-Ptests.badapples=false -Ptests.file.encoding=UTF-8 
-Ptests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/test-data/enwiki.random.lines.txt

  - org.apache.solr.hdfs.cloud.HdfsChaosMonkeyNothingIsSafeTest.test 
(:solr:modules:hdfs)
    Test history: 
https://ge.apache.org/scans/tests?search.rootProjectNames=solr-root&tests.container=org.apache.solr.hdfs.cloud.HdfsChaosMonkeyNothingIsSafeTest&tests.test=test
 
http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.hdfs.cloud.HdfsChaosMonkeyNothingIsSafeTest.test
    Test output: 
/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/solr/modules/hdfs/build/test-results/test/outputs/OUTPUT-org.apache.solr.hdfs.cloud.HdfsChaosMonkeyNothingIsSafeTest.txt
    Reproduce with: ./gradlew :solr:modules:hdfs:test --tests 
"org.apache.solr.hdfs.cloud.HdfsChaosMonkeyNothingIsSafeTest.test" 
-Ptests.jvms=4 -Ptests.haltonfailure=false 
"-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC 
-XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" 
-Ptests.seed=ADD8807A4E373E10 -Ptests.multiplier=2 -Ptests.nightly=true 
-Ptests.badapples=false -Ptests.file.encoding=UTF-8 
-Ptests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/test-data/enwiki.random.lines.txt

  - org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test (:solr:core)
    Test history: 
https://ge.apache.org/scans/tests?search.rootProjectNames=solr-root&tests.container=org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest&tests.test=test
 
http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test
    Test output: 
/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.txt
    Reproduce with: ./gradlew :solr:core:test --tests 
"org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test" -Ptests.jvms=4 
-Ptests.haltonfailure=false "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 
-XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" 
-Ptests.seed=ADD8807A4E373E10 -Ptests.multiplier=2 -Ptests.nightly=true 
-Ptests.badapples=false -Ptests.file.encoding=UTF-8 
-Ptests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/test-data/enwiki.random.lines.txt


Deprecated Gradle features were used in this build, making it incompatible with 
Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings 
and determine if they come from your own scripts or plugins.

For more on this, please refer to 
https://docs.gradle.org/8.4/userguide/command_line_interface.html#sec:command_line_warnings
 in the Gradle documentation.

BUILD SUCCESSFUL in 3h 3m 55s
154 actionable tasks: 154 executed

Publishing build scan...
https://ge.apache.org/s/ei4bdxpp5ibo2

Build step 'Invoke Gradle script' changed build result to SUCCESS
Archiving artifacts
‘**/*.events’ doesn’t match anything: ‘**’ exists but not ‘**/*.events’
No artifacts found that match the file pattern 
"**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
[Checks API] No suitable checks publisher found.
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to