[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 2100 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/2100/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

11 tests failed.
FAILED:  
org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple

Error Message:
Waiting for collection testSimple1 null Live Nodes: [127.0.0.1:33624_solr, 
127.0.0.1:55491_solr] Last available state: 
DocCollection(testSimple1//collections/testSimple1/state.json/21)={   
"pullReplicas":"0",   "replicationFactor":"2",   "shards":{     "shard1":{      
 "range":"80000000-ffffffff",       "state":"active",       "replicas":{        
 "core_node4":{           "core":"testSimple1_shard1_replica_n3",           
"base_url":"https://127.0.0.1:33624/solr";,           
"node_name":"127.0.0.1:33624_solr",           "state":"active",           
"type":"NRT",           "force_set_state":"false",           "leader":"true"},  
       "core_node12":{           "core":"testSimple1_shard1_replica_n11",       
    "base_url":"https://127.0.0.1:33624/solr";,           
"node_name":"127.0.0.1:33624_solr",           "state":"active",           
"type":"NRT",           "force_set_state":"false"}}},     "shard2":{       
"range":"0-7fffffff",       "state":"active",       "replicas":{         
"core_node6":{           "core":"testSimple1_shard2_replica_n5",           
"base_url":"https://127.0.0.1:33624/solr";,           
"node_name":"127.0.0.1:33624_solr",           "state":"active",           
"type":"NRT",           "force_set_state":"false",           "leader":"true"},  
       "core_node10":{           "core":"testSimple1_shard2_replica_n9",        
   "base_url":"https://127.0.0.1:38687/solr";,           
"node_name":"127.0.0.1:38687_solr",           "state":"down",           
"type":"NRT",           "force_set_state":"false"}}}},   
"router":{"name":"compositeId"},   "maxShardsPerNode":"2",   
"autoAddReplicas":"true",   "nrtReplicas":"2",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Waiting for collection testSimple1
null
Live Nodes: [127.0.0.1:33624_solr, 127.0.0.1:55491_solr]
Last available state: 
DocCollection(testSimple1//collections/testSimple1/state.json/21)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node4":{
          "core":"testSimple1_shard1_replica_n3",
          "base_url":"https://127.0.0.1:33624/solr";,
          "node_name":"127.0.0.1:33624_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"},
        "core_node12":{
          "core":"testSimple1_shard1_replica_n11",
          "base_url":"https://127.0.0.1:33624/solr";,
          "node_name":"127.0.0.1:33624_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false"}}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node6":{
          "core":"testSimple1_shard2_replica_n5",
          "base_url":"https://127.0.0.1:33624/solr";,
          "node_name":"127.0.0.1:33624_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"},
        "core_node10":{
          "core":"testSimple1_shard2_replica_n9",
          "base_url":"https://127.0.0.1:38687/solr";,
          "node_name":"127.0.0.1:38687_solr",
          "state":"down",
          "type":"NRT",
          "force_set_state":"false"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"true",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
        at 
__randomizedtesting.SeedInfo.seed([7584D7D24ABE3B7C:4D37F32C6D4DEFAD]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at 
org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:280)
        at 
org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple(AutoAddReplicasIntegrationTest.java:107)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple

Error Message:
Waiting for collection testSimple1 null Live Nodes: [127.0.0.1:55705_solr, 
127.0.0.1:65378_solr] Last available state: 
DocCollection(testSimple1//collections/testSimple1/state.json/22)={   
"pullReplicas":"0",   "replicationFactor":"2",   "shards":{     "shard1":{      
 "range":"80000000-ffffffff",       "state":"active",       "replicas":{        
 "core_node5":{           "core":"testSimple1_shard1_replica_n2",           
"base_url":"https://127.0.0.1:55705/solr";,           
"node_name":"127.0.0.1:55705_solr",           "state":"active",           
"type":"NRT",           "force_set_state":"false",           "leader":"true"},  
       "core_node12":{           "core":"testSimple1_shard1_replica_n11",       
    "base_url":"https://127.0.0.1:43290/solr";,           
"node_name":"127.0.0.1:43290_solr",           "state":"down",           
"type":"NRT",           "force_set_state":"false"}}},     "shard2":{       
"range":"0-7fffffff",       "state":"active",       "replicas":{         
"core_node7":{           "core":"testSimple1_shard2_replica_n4",           
"base_url":"https://127.0.0.1:55705/solr";,           
"node_name":"127.0.0.1:55705_solr",           "state":"active",           
"type":"NRT",           "force_set_state":"false",           "leader":"true"},  
       "core_node10":{           "core":"testSimple1_shard2_replica_n9",        
   "base_url":"https://127.0.0.1:55705/solr";,           
"node_name":"127.0.0.1:55705_solr",           "state":"active",           
"type":"NRT",           "force_set_state":"false"}}}},   
"router":{"name":"compositeId"},   "maxShardsPerNode":"2",   
"autoAddReplicas":"true",   "nrtReplicas":"2",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Waiting for collection testSimple1
null
Live Nodes: [127.0.0.1:55705_solr, 127.0.0.1:65378_solr]
Last available state: 
DocCollection(testSimple1//collections/testSimple1/state.json/22)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node5":{
          "core":"testSimple1_shard1_replica_n2",
          "base_url":"https://127.0.0.1:55705/solr";,
          "node_name":"127.0.0.1:55705_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"},
        "core_node12":{
          "core":"testSimple1_shard1_replica_n11",
          "base_url":"https://127.0.0.1:43290/solr";,
          "node_name":"127.0.0.1:43290_solr",
          "state":"down",
          "type":"NRT",
          "force_set_state":"false"}}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node7":{
          "core":"testSimple1_shard2_replica_n4",
          "base_url":"https://127.0.0.1:55705/solr";,
          "node_name":"127.0.0.1:55705_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"},
        "core_node10":{
          "core":"testSimple1_shard2_replica_n9",
          "base_url":"https://127.0.0.1:55705/solr";,
          "node_name":"127.0.0.1:55705_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"true",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
        at 
__randomizedtesting.SeedInfo.seed([7584D7D24ABE3B7C:4D37F32C6D4DEFAD]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at 
org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:280)
        at 
org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple(AutoAddReplicasIntegrationTest.java:107)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple

Error Message:
Waiting for collection testSimple1 null Live Nodes: [127.0.0.1:48571_solr, 
127.0.0.1:50853_solr] Last available state: 
DocCollection(testSimple1//collections/testSimple1/state.json/20)={   
"pullReplicas":"0",   "replicationFactor":"2",   "shards":{     "shard1":{      
 "range":"80000000-ffffffff",       "state":"active",       "replicas":{        
 "core_node3":{           "core":"testSimple1_shard1_replica_n1",           
"base_url":"https://127.0.0.1:50853/solr";,           
"node_name":"127.0.0.1:50853_solr",           "state":"active",           
"type":"NRT",           "force_set_state":"false",           "leader":"true"},  
       "core_node12":{           "core":"testSimple1_shard1_replica_n11",       
    "base_url":"https://127.0.0.1:46476/solr";,           
"node_name":"127.0.0.1:46476_solr",           "state":"down",           
"type":"NRT",           "force_set_state":"false"}}},     "shard2":{       
"range":"0-7fffffff",       "state":"active",       "replicas":{         
"core_node8":{           "core":"testSimple1_shard2_replica_n6",           
"base_url":"https://127.0.0.1:50853/solr";,           
"node_name":"127.0.0.1:50853_solr",           "state":"active",           
"type":"NRT",           "force_set_state":"false",           "leader":"true"},  
       "core_node10":{           "core":"testSimple1_shard2_replica_n9",        
   "base_url":"https://127.0.0.1:50853/solr";,           
"node_name":"127.0.0.1:50853_solr",           "state":"active",           
"type":"NRT",           "force_set_state":"false"}}}},   
"router":{"name":"compositeId"},   "maxShardsPerNode":"2",   
"autoAddReplicas":"true",   "nrtReplicas":"2",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Waiting for collection testSimple1
null
Live Nodes: [127.0.0.1:48571_solr, 127.0.0.1:50853_solr]
Last available state: 
DocCollection(testSimple1//collections/testSimple1/state.json/20)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node3":{
          "core":"testSimple1_shard1_replica_n1",
          "base_url":"https://127.0.0.1:50853/solr";,
          "node_name":"127.0.0.1:50853_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"},
        "core_node12":{
          "core":"testSimple1_shard1_replica_n11",
          "base_url":"https://127.0.0.1:46476/solr";,
          "node_name":"127.0.0.1:46476_solr",
          "state":"down",
          "type":"NRT",
          "force_set_state":"false"}}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node8":{
          "core":"testSimple1_shard2_replica_n6",
          "base_url":"https://127.0.0.1:50853/solr";,
          "node_name":"127.0.0.1:50853_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"},
        "core_node10":{
          "core":"testSimple1_shard2_replica_n9",
          "base_url":"https://127.0.0.1:50853/solr";,
          "node_name":"127.0.0.1:50853_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"true",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
        at 
__randomizedtesting.SeedInfo.seed([7584D7D24ABE3B7C:4D37F32C6D4DEFAD]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at 
org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:280)
        at 
org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple(AutoAddReplicasIntegrationTest.java:107)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple

Error Message:
Waiting for collection testSimple1 null Live Nodes: [127.0.0.1:54325_solr, 
127.0.0.1:59361_solr] Last available state: 
DocCollection(testSimple1//collections/testSimple1/state.json/20)={   
"pullReplicas":"0",   "replicationFactor":"2",   "shards":{     "shard1":{      
 "range":"80000000-ffffffff",       "state":"active",       "replicas":{        
 "core_node5":{           "core":"testSimple1_shard1_replica_n2",           
"base_url":"https://127.0.0.1:54325/solr";,           
"node_name":"127.0.0.1:54325_solr",           "state":"active",           
"type":"NRT",           "force_set_state":"false",           "leader":"true"},  
       "core_node12":{           "core":"testSimple1_shard1_replica_n11",       
    "base_url":"https://127.0.0.1:51107/solr";,           
"node_name":"127.0.0.1:51107_solr",           "state":"down",           
"type":"NRT",           "force_set_state":"false"}}},     "shard2":{       
"range":"0-7fffffff",       "state":"active",       "replicas":{         
"core_node7":{           "core":"testSimple1_shard2_replica_n4",           
"base_url":"https://127.0.0.1:54325/solr";,           
"node_name":"127.0.0.1:54325_solr",           "state":"active",           
"type":"NRT",           "force_set_state":"false",           "leader":"true"},  
       "core_node10":{           "core":"testSimple1_shard2_replica_n9",        
   "base_url":"https://127.0.0.1:54325/solr";,           
"node_name":"127.0.0.1:54325_solr",           "state":"active",           
"type":"NRT",           "force_set_state":"false"}}}},   
"router":{"name":"compositeId"},   "maxShardsPerNode":"2",   
"autoAddReplicas":"true",   "nrtReplicas":"2",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Waiting for collection testSimple1
null
Live Nodes: [127.0.0.1:54325_solr, 127.0.0.1:59361_solr]
Last available state: 
DocCollection(testSimple1//collections/testSimple1/state.json/20)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node5":{
          "core":"testSimple1_shard1_replica_n2",
          "base_url":"https://127.0.0.1:54325/solr";,
          "node_name":"127.0.0.1:54325_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"},
        "core_node12":{
          "core":"testSimple1_shard1_replica_n11",
          "base_url":"https://127.0.0.1:51107/solr";,
          "node_name":"127.0.0.1:51107_solr",
          "state":"down",
          "type":"NRT",
          "force_set_state":"false"}}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node7":{
          "core":"testSimple1_shard2_replica_n4",
          "base_url":"https://127.0.0.1:54325/solr";,
          "node_name":"127.0.0.1:54325_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"},
        "core_node10":{
          "core":"testSimple1_shard2_replica_n9",
          "base_url":"https://127.0.0.1:54325/solr";,
          "node_name":"127.0.0.1:54325_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"true",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
        at 
__randomizedtesting.SeedInfo.seed([7584D7D24ABE3B7C:4D37F32C6D4DEFAD]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at 
org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:280)
        at 
org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple(AutoAddReplicasIntegrationTest.java:107)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple

Error Message:
Waiting for collection testSimple1 null Live Nodes: [127.0.0.1:51653_solr, 
127.0.0.1:55109_solr] Last available state: 
DocCollection(testSimple1//collections/testSimple1/state.json/17)={   
"pullReplicas":"0",   "replicationFactor":"2",   "shards":{     "shard1":{      
 "range":"80000000-ffffffff",       "state":"active",       "replicas":{        
 "core_node5":{           "core":"testSimple1_shard1_replica_n2",           
"base_url":"https://127.0.0.1:51653/solr";,           
"node_name":"127.0.0.1:51653_solr",           "state":"active",           
"type":"NRT",           "force_set_state":"false",           "leader":"true"},  
       "core_node12":{           "core":"testSimple1_shard1_replica_n11",       
    "base_url":"https://127.0.0.1:51653/solr";,           
"node_name":"127.0.0.1:51653_solr",           "state":"active",           
"type":"NRT",           "force_set_state":"false"}}},     "shard2":{       
"range":"0-7fffffff",       "state":"active",       "replicas":{         
"core_node6":{           "core":"testSimple1_shard2_replica_n4",           
"base_url":"https://127.0.0.1:51653/solr";,           
"node_name":"127.0.0.1:51653_solr",           "state":"active",           
"type":"NRT",           "force_set_state":"false",           "leader":"true"},  
       "core_node10":{           "core":"testSimple1_shard2_replica_n9",        
   "base_url":"https://127.0.0.1:52974/solr";,           
"node_name":"127.0.0.1:52974_solr",           "state":"down",           
"type":"NRT",           "force_set_state":"false"}}}},   
"router":{"name":"compositeId"},   "maxShardsPerNode":"2",   
"autoAddReplicas":"true",   "nrtReplicas":"2",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Waiting for collection testSimple1
null
Live Nodes: [127.0.0.1:51653_solr, 127.0.0.1:55109_solr]
Last available state: 
DocCollection(testSimple1//collections/testSimple1/state.json/17)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node5":{
          "core":"testSimple1_shard1_replica_n2",
          "base_url":"https://127.0.0.1:51653/solr";,
          "node_name":"127.0.0.1:51653_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"},
        "core_node12":{
          "core":"testSimple1_shard1_replica_n11",
          "base_url":"https://127.0.0.1:51653/solr";,
          "node_name":"127.0.0.1:51653_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false"}}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node6":{
          "core":"testSimple1_shard2_replica_n4",
          "base_url":"https://127.0.0.1:51653/solr";,
          "node_name":"127.0.0.1:51653_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"},
        "core_node10":{
          "core":"testSimple1_shard2_replica_n9",
          "base_url":"https://127.0.0.1:52974/solr";,
          "node_name":"127.0.0.1:52974_solr",
          "state":"down",
          "type":"NRT",
          "force_set_state":"false"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"true",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
        at 
__randomizedtesting.SeedInfo.seed([7584D7D24ABE3B7C:4D37F32C6D4DEFAD]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at 
org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:280)
        at 
org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple(AutoAddReplicasIntegrationTest.java:107)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest.testSimple

Error Message:
Waiting for collection testSimple1 null Live Nodes: [127.0.0.1:57391_solr, 
127.0.0.1:58784_solr] Last available state: 
DocCollection(testSimple1//collections/testSimple1/state.json/17)={   
"pullReplicas":"0",   "replicationFactor":"2",   "shards":{     "shard1":{      
 "range":"80000000-ffffffff",       "state":"active",       "replicas":{        
 "core_node3":{           
"dataDir":"hdfs://serv1-vm3:60518/data/testSimple1/core_node3/data/",           
"base_url":"http://127.0.0.1:57391/solr";,           
"node_name":"127.0.0.1:57391_solr",           "type":"NRT",           
"force_set_state":"false",           
"ulogDir":"hdfs://serv1-vm3:60518/data/testSimple1/core_node3/data/tlog",       
    "core":"testSimple1_shard1_replica_n1",           "shared_storage":"true",  
         "state":"active",           "leader":"true"},         "core_node5":{   
        "dataDir":"hdfs://serv1-vm3:60518/data/testSimple1/core_node5/data/",   
        "base_url":"http://127.0.0.1:39854/solr";,           
"node_name":"127.0.0.1:39854_solr",           "type":"NRT",           
"force_set_state":"false",           
"ulogDir":"hdfs://serv1-vm3:60518/data/testSimple1/core_node5/data/tlog",       
    "core":"testSimple1_shard1_replica_n2",           "shared_storage":"true",  
         "state":"down"}}},     "shard2":{       "range":"0-7fffffff",       
"state":"active",       "replicas":{         "core_node7":{           
"dataDir":"hdfs://serv1-vm3:60518/data/testSimple1/core_node7/data/",           
"base_url":"http://127.0.0.1:57391/solr";,           
"node_name":"127.0.0.1:57391_solr",           "type":"NRT",           
"force_set_state":"false",           
"ulogDir":"hdfs://serv1-vm3:60518/data/testSimple1/core_node7/data/tlog",       
    "core":"testSimple1_shard2_replica_n4",           "shared_storage":"true",  
         "state":"active"},         "core_node8":{           
"dataDir":"hdfs://serv1-vm3:60518/data/testSimple1/core_node8/data/",           
"base_url":"http://127.0.0.1:57391/solr";,           
"node_name":"127.0.0.1:57391_solr",           "type":"NRT",           
"force_set_state":"false",           
"ulogDir":"hdfs://serv1-vm3:60518/data/testSimple1/core_node8/data/tlog",       
    "core":"testSimple1_shard2_replica_n6",           "shared_storage":"true",  
         "state":"active",           "leader":"true"}}}},   
"router":{"name":"compositeId"},   "maxShardsPerNode":"2",   
"autoAddReplicas":"true",   "nrtReplicas":"2",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Waiting for collection testSimple1
null
Live Nodes: [127.0.0.1:57391_solr, 127.0.0.1:58784_solr]
Last available state: 
DocCollection(testSimple1//collections/testSimple1/state.json/17)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node3":{
          "dataDir":"hdfs://serv1-vm3:60518/data/testSimple1/core_node3/data/",
          "base_url":"http://127.0.0.1:57391/solr";,
          "node_name":"127.0.0.1:57391_solr",
          "type":"NRT",
          "force_set_state":"false",
          
"ulogDir":"hdfs://serv1-vm3:60518/data/testSimple1/core_node3/data/tlog",
          "core":"testSimple1_shard1_replica_n1",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"},
        "core_node5":{
          "dataDir":"hdfs://serv1-vm3:60518/data/testSimple1/core_node5/data/",
          "base_url":"http://127.0.0.1:39854/solr";,
          "node_name":"127.0.0.1:39854_solr",
          "type":"NRT",
          "force_set_state":"false",
          
"ulogDir":"hdfs://serv1-vm3:60518/data/testSimple1/core_node5/data/tlog",
          "core":"testSimple1_shard1_replica_n2",
          "shared_storage":"true",
          "state":"down"}}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node7":{
          "dataDir":"hdfs://serv1-vm3:60518/data/testSimple1/core_node7/data/",
          "base_url":"http://127.0.0.1:57391/solr";,
          "node_name":"127.0.0.1:57391_solr",
          "type":"NRT",
          "force_set_state":"false",
          
"ulogDir":"hdfs://serv1-vm3:60518/data/testSimple1/core_node7/data/tlog",
          "core":"testSimple1_shard2_replica_n4",
          "shared_storage":"true",
          "state":"active"},
        "core_node8":{
          "dataDir":"hdfs://serv1-vm3:60518/data/testSimple1/core_node8/data/",
          "base_url":"http://127.0.0.1:57391/solr";,
          "node_name":"127.0.0.1:57391_solr",
          "type":"NRT",
          "force_set_state":"false",
          
"ulogDir":"hdfs://serv1-vm3:60518/data/testSimple1/core_node8/data/tlog",
          "core":"testSimple1_shard2_replica_n6",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"true",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
        at 
__randomizedtesting.SeedInfo.seed([7584D7D24ABE3B7C:4D37F32C6D4DEFAD]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at 
org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:280)
        at 
org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple(AutoAddReplicasIntegrationTest.java:107)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest.testSimple

Error Message:
Waiting for collection testSimple1 null Live Nodes: [127.0.0.1:43206_solr, 
127.0.0.1:50792_solr] Last available state: 
DocCollection(testSimple1//collections/testSimple1/state.json/17)={   
"pullReplicas":"0",   "replicationFactor":"2",   "shards":{     "shard1":{      
 "range":"80000000-ffffffff",       "state":"active",       "replicas":{        
 "core_node3":{           
"dataDir":"hdfs://serv1-vm3:42396/data/testSimple1/core_node3/data/",           
"base_url":"http://127.0.0.1:50792/solr";,           
"node_name":"127.0.0.1:50792_solr",           "type":"NRT",           
"force_set_state":"false",           
"ulogDir":"hdfs://serv1-vm3:42396/data/testSimple1/core_node3/data/tlog",       
    "core":"testSimple1_shard1_replica_n1",           "shared_storage":"true",  
         "state":"active",           "leader":"true"},         "core_node5":{   
        "dataDir":"hdfs://serv1-vm3:42396/data/testSimple1/core_node5/data/",   
        "base_url":"http://127.0.0.1:32975/solr";,           
"node_name":"127.0.0.1:32975_solr",           "type":"NRT",           
"force_set_state":"false",           
"ulogDir":"hdfs://serv1-vm3:42396/data/testSimple1/core_node5/data/tlog",       
    "core":"testSimple1_shard1_replica_n2",           "shared_storage":"true",  
         "state":"down"}}},     "shard2":{       "range":"0-7fffffff",       
"state":"active",       "replicas":{         "core_node7":{           
"dataDir":"hdfs://serv1-vm3:42396/data/testSimple1/core_node7/data/",           
"base_url":"http://127.0.0.1:50792/solr";,           
"node_name":"127.0.0.1:50792_solr",           "type":"NRT",           
"force_set_state":"false",           
"ulogDir":"hdfs://serv1-vm3:42396/data/testSimple1/core_node7/data/tlog",       
    "core":"testSimple1_shard2_replica_n4",           "shared_storage":"true",  
         "state":"active"},         "core_node8":{           
"dataDir":"hdfs://serv1-vm3:42396/data/testSimple1/core_node8/data/",           
"base_url":"http://127.0.0.1:50792/solr";,           
"node_name":"127.0.0.1:50792_solr",           "type":"NRT",           
"force_set_state":"false",           
"ulogDir":"hdfs://serv1-vm3:42396/data/testSimple1/core_node8/data/tlog",       
    "core":"testSimple1_shard2_replica_n6",           "shared_storage":"true",  
         "state":"active",           "leader":"true"}}}},   
"router":{"name":"compositeId"},   "maxShardsPerNode":"2",   
"autoAddReplicas":"true",   "nrtReplicas":"2",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Waiting for collection testSimple1
null
Live Nodes: [127.0.0.1:43206_solr, 127.0.0.1:50792_solr]
Last available state: 
DocCollection(testSimple1//collections/testSimple1/state.json/17)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node3":{
          "dataDir":"hdfs://serv1-vm3:42396/data/testSimple1/core_node3/data/",
          "base_url":"http://127.0.0.1:50792/solr";,
          "node_name":"127.0.0.1:50792_solr",
          "type":"NRT",
          "force_set_state":"false",
          
"ulogDir":"hdfs://serv1-vm3:42396/data/testSimple1/core_node3/data/tlog",
          "core":"testSimple1_shard1_replica_n1",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"},
        "core_node5":{
          "dataDir":"hdfs://serv1-vm3:42396/data/testSimple1/core_node5/data/",
          "base_url":"http://127.0.0.1:32975/solr";,
          "node_name":"127.0.0.1:32975_solr",
          "type":"NRT",
          "force_set_state":"false",
          
"ulogDir":"hdfs://serv1-vm3:42396/data/testSimple1/core_node5/data/tlog",
          "core":"testSimple1_shard1_replica_n2",
          "shared_storage":"true",
          "state":"down"}}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node7":{
          "dataDir":"hdfs://serv1-vm3:42396/data/testSimple1/core_node7/data/",
          "base_url":"http://127.0.0.1:50792/solr";,
          "node_name":"127.0.0.1:50792_solr",
          "type":"NRT",
          "force_set_state":"false",
          
"ulogDir":"hdfs://serv1-vm3:42396/data/testSimple1/core_node7/data/tlog",
          "core":"testSimple1_shard2_replica_n4",
          "shared_storage":"true",
          "state":"active"},
        "core_node8":{
          "dataDir":"hdfs://serv1-vm3:42396/data/testSimple1/core_node8/data/",
          "base_url":"http://127.0.0.1:50792/solr";,
          "node_name":"127.0.0.1:50792_solr",
          "type":"NRT",
          "force_set_state":"false",
          
"ulogDir":"hdfs://serv1-vm3:42396/data/testSimple1/core_node8/data/tlog",
          "core":"testSimple1_shard2_replica_n6",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"true",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
        at 
__randomizedtesting.SeedInfo.seed([7584D7D24ABE3B7C:4D37F32C6D4DEFAD]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at 
org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:280)
        at 
org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple(AutoAddReplicasIntegrationTest.java:107)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest.testSimple

Error Message:
Waiting for collection testSimple1 null Live Nodes: [127.0.0.1:53277_solr, 
127.0.0.1:58388_solr] Last available state: 
DocCollection(testSimple1//collections/testSimple1/state.json/18)={   
"pullReplicas":"0",   "replicationFactor":"2",   "shards":{     "shard1":{      
 "range":"80000000-ffffffff",       "state":"active",       "replicas":{        
 "core_node3":{           
"dataDir":"hdfs://serv1-vm3:34322/data/testSimple1/core_node3/data/",           
"base_url":"http://127.0.0.1:53277/solr";,           
"node_name":"127.0.0.1:53277_solr",           "type":"NRT",           
"force_set_state":"false",           
"ulogDir":"hdfs://serv1-vm3:34322/data/testSimple1/core_node3/data/tlog",       
    "core":"testSimple1_shard1_replica_n1",           "shared_storage":"true",  
         "state":"active"},         "core_node5":{           
"dataDir":"hdfs://serv1-vm3:34322/data/testSimple1/core_node5/data/",           
"base_url":"http://127.0.0.1:53277/solr";,           
"node_name":"127.0.0.1:53277_solr",           "type":"NRT",           
"force_set_state":"false",           
"ulogDir":"hdfs://serv1-vm3:34322/data/testSimple1/core_node5/data/tlog",       
    "core":"testSimple1_shard1_replica_n2",           "shared_storage":"true",  
         "state":"active",           "leader":"true"}}},     "shard2":{       
"range":"0-7fffffff",       "state":"active",       "replicas":{         
"core_node7":{           
"dataDir":"hdfs://serv1-vm3:34322/data/testSimple1/core_node7/data/",           
"base_url":"http://127.0.0.1:53277/solr";,           
"node_name":"127.0.0.1:53277_solr",           "type":"NRT",           
"force_set_state":"false",           
"ulogDir":"hdfs://serv1-vm3:34322/data/testSimple1/core_node7/data/tlog",       
    "core":"testSimple1_shard2_replica_n4",           "shared_storage":"true",  
         "state":"active",           "leader":"true"},         "core_node8":{   
        "dataDir":"hdfs://serv1-vm3:34322/data/testSimple1/core_node8/data/",   
        "base_url":"http://127.0.0.1:59511/solr";,           
"node_name":"127.0.0.1:59511_solr",           "type":"NRT",           
"force_set_state":"false",           
"ulogDir":"hdfs://serv1-vm3:34322/data/testSimple1/core_node8/data/tlog",       
    "core":"testSimple1_shard2_replica_n6",           "shared_storage":"true",  
         "state":"down"}}}},   "router":{"name":"compositeId"},   
"maxShardsPerNode":"2",   "autoAddReplicas":"true",   "nrtReplicas":"2",   
"tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Waiting for collection testSimple1
null
Live Nodes: [127.0.0.1:53277_solr, 127.0.0.1:58388_solr]
Last available state: 
DocCollection(testSimple1//collections/testSimple1/state.json/18)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node3":{
          "dataDir":"hdfs://serv1-vm3:34322/data/testSimple1/core_node3/data/",
          "base_url":"http://127.0.0.1:53277/solr";,
          "node_name":"127.0.0.1:53277_solr",
          "type":"NRT",
          "force_set_state":"false",
          
"ulogDir":"hdfs://serv1-vm3:34322/data/testSimple1/core_node3/data/tlog",
          "core":"testSimple1_shard1_replica_n1",
          "shared_storage":"true",
          "state":"active"},
        "core_node5":{
          "dataDir":"hdfs://serv1-vm3:34322/data/testSimple1/core_node5/data/",
          "base_url":"http://127.0.0.1:53277/solr";,
          "node_name":"127.0.0.1:53277_solr",
          "type":"NRT",
          "force_set_state":"false",
          
"ulogDir":"hdfs://serv1-vm3:34322/data/testSimple1/core_node5/data/tlog",
          "core":"testSimple1_shard1_replica_n2",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"}}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node7":{
          "dataDir":"hdfs://serv1-vm3:34322/data/testSimple1/core_node7/data/",
          "base_url":"http://127.0.0.1:53277/solr";,
          "node_name":"127.0.0.1:53277_solr",
          "type":"NRT",
          "force_set_state":"false",
          
"ulogDir":"hdfs://serv1-vm3:34322/data/testSimple1/core_node7/data/tlog",
          "core":"testSimple1_shard2_replica_n4",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"},
        "core_node8":{
          "dataDir":"hdfs://serv1-vm3:34322/data/testSimple1/core_node8/data/",
          "base_url":"http://127.0.0.1:59511/solr";,
          "node_name":"127.0.0.1:59511_solr",
          "type":"NRT",
          "force_set_state":"false",
          
"ulogDir":"hdfs://serv1-vm3:34322/data/testSimple1/core_node8/data/tlog",
          "core":"testSimple1_shard2_replica_n6",
          "shared_storage":"true",
          "state":"down"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"true",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
        at 
__randomizedtesting.SeedInfo.seed([7584D7D24ABE3B7C:4D37F32C6D4DEFAD]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at 
org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:280)
        at 
org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple(AutoAddReplicasIntegrationTest.java:107)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest.testSimple

Error Message:
Waiting for collection testSimple1 null Live Nodes: [127.0.0.1:44559_solr, 
127.0.0.1:62661_solr] Last available state: 
DocCollection(testSimple1//collections/testSimple1/state.json/18)={   
"pullReplicas":"0",   "replicationFactor":"2",   "shards":{     "shard1":{      
 "range":"80000000-ffffffff",       "state":"active",       "replicas":{        
 "core_node3":{           
"dataDir":"hdfs://serv1-vm3:45886/data/testSimple1/core_node3/data/",           
"base_url":"http://127.0.0.1:44559/solr";,           
"node_name":"127.0.0.1:44559_solr",           "type":"NRT",           
"force_set_state":"false",           
"ulogDir":"hdfs://serv1-vm3:45886/data/testSimple1/core_node3/data/tlog",       
    "core":"testSimple1_shard1_replica_n1",           "shared_storage":"true",  
         "state":"active",           "leader":"true"},         "core_node5":{   
        "dataDir":"hdfs://serv1-vm3:45886/data/testSimple1/core_node5/data/",   
        "base_url":"http://127.0.0.1:44559/solr";,           
"node_name":"127.0.0.1:44559_solr",           "type":"NRT",           
"force_set_state":"false",           
"ulogDir":"hdfs://serv1-vm3:45886/data/testSimple1/core_node5/data/tlog",       
    "core":"testSimple1_shard1_replica_n2",           "shared_storage":"true",  
         "state":"active"}}},     "shard2":{       "range":"0-7fffffff",       
"state":"active",       "replicas":{         "core_node7":{           
"dataDir":"hdfs://serv1-vm3:45886/data/testSimple1/core_node7/data/",           
"base_url":"http://127.0.0.1:51320/solr";,           
"node_name":"127.0.0.1:51320_solr",           "type":"NRT",           
"force_set_state":"false",           
"ulogDir":"hdfs://serv1-vm3:45886/data/testSimple1/core_node7/data/tlog",       
    "core":"testSimple1_shard2_replica_n4",           "shared_storage":"true",  
         "state":"down"},         "core_node8":{           
"dataDir":"hdfs://serv1-vm3:45886/data/testSimple1/core_node8/data/",           
"base_url":"http://127.0.0.1:44559/solr";,           
"node_name":"127.0.0.1:44559_solr",           "type":"NRT",           
"force_set_state":"false",           
"ulogDir":"hdfs://serv1-vm3:45886/data/testSimple1/core_node8/data/tlog",       
    "core":"testSimple1_shard2_replica_n6",           "shared_storage":"true",  
         "state":"active",           "leader":"true"}}}},   
"router":{"name":"compositeId"},   "maxShardsPerNode":"2",   
"autoAddReplicas":"true",   "nrtReplicas":"2",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Waiting for collection testSimple1
null
Live Nodes: [127.0.0.1:44559_solr, 127.0.0.1:62661_solr]
Last available state: 
DocCollection(testSimple1//collections/testSimple1/state.json/18)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node3":{
          "dataDir":"hdfs://serv1-vm3:45886/data/testSimple1/core_node3/data/",
          "base_url":"http://127.0.0.1:44559/solr";,
          "node_name":"127.0.0.1:44559_solr",
          "type":"NRT",
          "force_set_state":"false",
          
"ulogDir":"hdfs://serv1-vm3:45886/data/testSimple1/core_node3/data/tlog",
          "core":"testSimple1_shard1_replica_n1",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"},
        "core_node5":{
          "dataDir":"hdfs://serv1-vm3:45886/data/testSimple1/core_node5/data/",
          "base_url":"http://127.0.0.1:44559/solr";,
          "node_name":"127.0.0.1:44559_solr",
          "type":"NRT",
          "force_set_state":"false",
          
"ulogDir":"hdfs://serv1-vm3:45886/data/testSimple1/core_node5/data/tlog",
          "core":"testSimple1_shard1_replica_n2",
          "shared_storage":"true",
          "state":"active"}}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node7":{
          "dataDir":"hdfs://serv1-vm3:45886/data/testSimple1/core_node7/data/",
          "base_url":"http://127.0.0.1:51320/solr";,
          "node_name":"127.0.0.1:51320_solr",
          "type":"NRT",
          "force_set_state":"false",
          
"ulogDir":"hdfs://serv1-vm3:45886/data/testSimple1/core_node7/data/tlog",
          "core":"testSimple1_shard2_replica_n4",
          "shared_storage":"true",
          "state":"down"},
        "core_node8":{
          "dataDir":"hdfs://serv1-vm3:45886/data/testSimple1/core_node8/data/",
          "base_url":"http://127.0.0.1:44559/solr";,
          "node_name":"127.0.0.1:44559_solr",
          "type":"NRT",
          "force_set_state":"false",
          
"ulogDir":"hdfs://serv1-vm3:45886/data/testSimple1/core_node8/data/tlog",
          "core":"testSimple1_shard2_replica_n6",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"true",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
        at 
__randomizedtesting.SeedInfo.seed([7584D7D24ABE3B7C:4D37F32C6D4DEFAD]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at 
org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:280)
        at 
org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple(AutoAddReplicasIntegrationTest.java:107)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest.testSimple

Error Message:
Waiting for collection testSimple1 null Live Nodes: [127.0.0.1:52266_solr, 
127.0.0.1:54226_solr] Last available state: 
DocCollection(testSimple1//collections/testSimple1/state.json/14)={   
"pullReplicas":"0",   "replicationFactor":"2",   "shards":{     "shard1":{      
 "range":"80000000-ffffffff",       "state":"active",       "replicas":{        
 "core_node3":{           
"dataDir":"hdfs://serv1-vm3:63352/data/testSimple1/core_node3/data/",           
"base_url":"http://127.0.0.1:54226/solr";,           
"node_name":"127.0.0.1:54226_solr",           "type":"NRT",           
"force_set_state":"false",           
"ulogDir":"hdfs://serv1-vm3:63352/data/testSimple1/core_node3/data/tlog",       
    "core":"testSimple1_shard1_replica_n1",           "shared_storage":"true",  
         "state":"active"},         "core_node5":{           
"dataDir":"hdfs://serv1-vm3:63352/data/testSimple1/core_node5/data/",           
"base_url":"http://127.0.0.1:54226/solr";,           
"node_name":"127.0.0.1:54226_solr",           "type":"NRT",           
"force_set_state":"false",           
"ulogDir":"hdfs://serv1-vm3:63352/data/testSimple1/core_node5/data/tlog",       
    "core":"testSimple1_shard1_replica_n2",           "shared_storage":"true",  
         "state":"active",           "leader":"true"}}},     "shard2":{       
"range":"0-7fffffff",       "state":"active",       "replicas":{         
"core_node7":{           
"dataDir":"hdfs://serv1-vm3:63352/data/testSimple1/core_node7/data/",           
"base_url":"http://127.0.0.1:54226/solr";,           
"node_name":"127.0.0.1:54226_solr",           "type":"NRT",           
"force_set_state":"false",           
"ulogDir":"hdfs://serv1-vm3:63352/data/testSimple1/core_node7/data/tlog",       
    "core":"testSimple1_shard2_replica_n4",           "shared_storage":"true",  
         "state":"active",           "leader":"true"},         "core_node8":{   
        "dataDir":"hdfs://serv1-vm3:63352/data/testSimple1/core_node8/data/",   
        "base_url":"http://127.0.0.1:56156/solr";,           
"node_name":"127.0.0.1:56156_solr",           "type":"NRT",           
"force_set_state":"false",           
"ulogDir":"hdfs://serv1-vm3:63352/data/testSimple1/core_node8/data/tlog",       
    "core":"testSimple1_shard2_replica_n6",           "shared_storage":"true",  
         "state":"down"}}}},   "router":{"name":"compositeId"},   
"maxShardsPerNode":"2",   "autoAddReplicas":"true",   "nrtReplicas":"2",   
"tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Waiting for collection testSimple1
null
Live Nodes: [127.0.0.1:52266_solr, 127.0.0.1:54226_solr]
Last available state: 
DocCollection(testSimple1//collections/testSimple1/state.json/14)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node3":{
          "dataDir":"hdfs://serv1-vm3:63352/data/testSimple1/core_node3/data/",
          "base_url":"http://127.0.0.1:54226/solr";,
          "node_name":"127.0.0.1:54226_solr",
          "type":"NRT",
          "force_set_state":"false",
          
"ulogDir":"hdfs://serv1-vm3:63352/data/testSimple1/core_node3/data/tlog",
          "core":"testSimple1_shard1_replica_n1",
          "shared_storage":"true",
          "state":"active"},
        "core_node5":{
          "dataDir":"hdfs://serv1-vm3:63352/data/testSimple1/core_node5/data/",
          "base_url":"http://127.0.0.1:54226/solr";,
          "node_name":"127.0.0.1:54226_solr",
          "type":"NRT",
          "force_set_state":"false",
          
"ulogDir":"hdfs://serv1-vm3:63352/data/testSimple1/core_node5/data/tlog",
          "core":"testSimple1_shard1_replica_n2",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"}}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node7":{
          "dataDir":"hdfs://serv1-vm3:63352/data/testSimple1/core_node7/data/",
          "base_url":"http://127.0.0.1:54226/solr";,
          "node_name":"127.0.0.1:54226_solr",
          "type":"NRT",
          "force_set_state":"false",
          
"ulogDir":"hdfs://serv1-vm3:63352/data/testSimple1/core_node7/data/tlog",
          "core":"testSimple1_shard2_replica_n4",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"},
        "core_node8":{
          "dataDir":"hdfs://serv1-vm3:63352/data/testSimple1/core_node8/data/",
          "base_url":"http://127.0.0.1:56156/solr";,
          "node_name":"127.0.0.1:56156_solr",
          "type":"NRT",
          "force_set_state":"false",
          
"ulogDir":"hdfs://serv1-vm3:63352/data/testSimple1/core_node8/data/tlog",
          "core":"testSimple1_shard2_replica_n6",
          "shared_storage":"true",
          "state":"down"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"true",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
        at 
__randomizedtesting.SeedInfo.seed([7584D7D24ABE3B7C:4D37F32C6D4DEFAD]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at 
org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:280)
        at 
org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple(AutoAddReplicasIntegrationTest.java:107)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest.testSimple

Error Message:
Waiting for collection testSimple1 null Live Nodes: [127.0.0.1:43358_solr, 
127.0.0.1:51139_solr] Last available state: 
DocCollection(testSimple1//collections/testSimple1/state.json/16)={   
"pullReplicas":"0",   "replicationFactor":"2",   "shards":{     "shard1":{      
 "range":"80000000-ffffffff",       "state":"active",       "replicas":{        
 "core_node3":{           
"dataDir":"hdfs://serv1-vm3:51852/data/testSimple1/core_node3/data/",           
"base_url":"http://127.0.0.1:43358/solr";,           
"node_name":"127.0.0.1:43358_solr",           "type":"NRT",           
"force_set_state":"false",           
"ulogDir":"hdfs://serv1-vm3:51852/data/testSimple1/core_node3/data/tlog",       
    "core":"testSimple1_shard1_replica_n1",           "shared_storage":"true",  
         "state":"active"},         "core_node5":{           
"dataDir":"hdfs://serv1-vm3:51852/data/testSimple1/core_node5/data/",           
"base_url":"http://127.0.0.1:43358/solr";,           
"node_name":"127.0.0.1:43358_solr",           "type":"NRT",           
"force_set_state":"false",           
"ulogDir":"hdfs://serv1-vm3:51852/data/testSimple1/core_node5/data/tlog",       
    "core":"testSimple1_shard1_replica_n2",           "shared_storage":"true",  
         "state":"active",           "leader":"true"}}},     "shard2":{       
"range":"0-7fffffff",       "state":"active",       "replicas":{         
"core_node7":{           
"dataDir":"hdfs://serv1-vm3:51852/data/testSimple1/core_node7/data/",           
"base_url":"http://127.0.0.1:43358/solr";,           
"node_name":"127.0.0.1:43358_solr",           "type":"NRT",           
"force_set_state":"false",           
"ulogDir":"hdfs://serv1-vm3:51852/data/testSimple1/core_node7/data/tlog",       
    "core":"testSimple1_shard2_replica_n4",           "shared_storage":"true",  
         "state":"active",           "leader":"true"},         "core_node8":{   
        "dataDir":"hdfs://serv1-vm3:51852/data/testSimple1/core_node8/data/",   
        "base_url":"http://127.0.0.1:49607/solr";,           
"node_name":"127.0.0.1:49607_solr",           "type":"NRT",           
"force_set_state":"false",           
"ulogDir":"hdfs://serv1-vm3:51852/data/testSimple1/core_node8/data/tlog",       
    "core":"testSimple1_shard2_replica_n6",           "shared_storage":"true",  
         "state":"down"}}}},   "router":{"name":"compositeId"},   
"maxShardsPerNode":"2",   "autoAddReplicas":"true",   "nrtReplicas":"2",   
"tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Waiting for collection testSimple1
null
Live Nodes: [127.0.0.1:43358_solr, 127.0.0.1:51139_solr]
Last available state: 
DocCollection(testSimple1//collections/testSimple1/state.json/16)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node3":{
          "dataDir":"hdfs://serv1-vm3:51852/data/testSimple1/core_node3/data/",
          "base_url":"http://127.0.0.1:43358/solr";,
          "node_name":"127.0.0.1:43358_solr",
          "type":"NRT",
          "force_set_state":"false",
          
"ulogDir":"hdfs://serv1-vm3:51852/data/testSimple1/core_node3/data/tlog",
          "core":"testSimple1_shard1_replica_n1",
          "shared_storage":"true",
          "state":"active"},
        "core_node5":{
          "dataDir":"hdfs://serv1-vm3:51852/data/testSimple1/core_node5/data/",
          "base_url":"http://127.0.0.1:43358/solr";,
          "node_name":"127.0.0.1:43358_solr",
          "type":"NRT",
          "force_set_state":"false",
          
"ulogDir":"hdfs://serv1-vm3:51852/data/testSimple1/core_node5/data/tlog",
          "core":"testSimple1_shard1_replica_n2",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"}}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node7":{
          "dataDir":"hdfs://serv1-vm3:51852/data/testSimple1/core_node7/data/",
          "base_url":"http://127.0.0.1:43358/solr";,
          "node_name":"127.0.0.1:43358_solr",
          "type":"NRT",
          "force_set_state":"false",
          
"ulogDir":"hdfs://serv1-vm3:51852/data/testSimple1/core_node7/data/tlog",
          "core":"testSimple1_shard2_replica_n4",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"},
        "core_node8":{
          "dataDir":"hdfs://serv1-vm3:51852/data/testSimple1/core_node8/data/",
          "base_url":"http://127.0.0.1:49607/solr";,
          "node_name":"127.0.0.1:49607_solr",
          "type":"NRT",
          "force_set_state":"false",
          
"ulogDir":"hdfs://serv1-vm3:51852/data/testSimple1/core_node8/data/tlog",
          "core":"testSimple1_shard2_replica_n6",
          "shared_storage":"true",
          "state":"down"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"true",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
        at 
__randomizedtesting.SeedInfo.seed([7584D7D24ABE3B7C:4D37F32C6D4DEFAD]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at 
org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:280)
        at 
org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple(AutoAddReplicasIntegrationTest.java:107)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.eva

[...truncated too long message...]

ting to 127.0.0.1:46965 46965
   [junit4]   2> 682361 WARN  (Thread-626) [    ] o.a.s.c.ZkTestServer Watch 
limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2>        9       /solr/clusterprops.json
   [junit4]   2>        9       /solr/aliases.json
   [junit4]   2>        6       /solr/collections/testSimple1/terms/shard2
   [junit4]   2>        5       /solr/collections/testSimple2/terms/shard2
   [junit4]   2>        5       /solr/collections/testSimple1/terms/shard1
   [junit4]   2>        4       /solr/security.json
   [junit4]   2>        4       /solr/configs/conf
   [junit4]   2>        3       /solr/collections/testSimple2/terms/shard1
   [junit4]   2>        3       /solr/collections/testSimple3/terms/shard3
   [junit4]   2>        3       /solr/collections/testSimple3/terms/shard2
   [junit4]   2>        3       /solr/collections/testSimple3/terms/shard1
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2>        19      /solr/collections/testSimple3/state.json
   [junit4]   2>        14      /solr/collections/testSimple1/state.json
   [junit4]   2>        9       /solr/clusterstate.json
   [junit4]   2>        7       /solr/clusterprops.json
   [junit4]   2>        4       /solr/collections/testSimple2/state.json
   [junit4]   2>        2       
/solr/overseer_elect/election/72071079620116488-127.0.0.1:44559_solr-n_0000000001
   [junit4]   2>        2       /solr/autoscaling.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2>        9       /solr/live_nodes
   [junit4]   2>        9       /solr/collections
   [junit4]   2>        2       /solr/overseer/queue
   [junit4]   2>        2       /solr/autoscaling/events/.scheduled_maintenance
   [junit4]   2>        2       /solr/autoscaling/events/.auto_add_replicas
   [junit4]   2>        2       /solr/overseer/queue-work
   [junit4]   2>        2       /solr/overseer/collection-queue-work
   [junit4]   2> 
   [junit4]   2> 682367 WARN  
(SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[7584D7D24ABE3B7C]-worker) [    
] o.a.h.h.s.d.DirectoryScanner DirectoryScanner: shutdown has been called
   [junit4]   2> 682375 INFO  
(SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[7584D7D24ABE3B7C]-worker) [    
] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
   [junit4]   2> 682477 WARN  (DataNode: 
[[[DISK]file:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_7584D7D24ABE3B7C-003/tempDir-002/hdfsBaseDir/data/data3/,
 
[DISK]file:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_7584D7D24ABE3B7C-003/tempDir-002/hdfsBaseDir/data/data4/]]
  heartbeating to serv1-vm3/127.0.0.1:45886) [    ] 
o.a.h.h.s.d.IncrementalBlockReportManager IncrementalBlockReportManager 
interrupted
   [junit4]   2> 682477 WARN  (DataNode: 
[[[DISK]file:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_7584D7D24ABE3B7C-003/tempDir-002/hdfsBaseDir/data/data3/,
 
[DISK]file:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_7584D7D24ABE3B7C-003/tempDir-002/hdfsBaseDir/data/data4/]]
  heartbeating to serv1-vm3/127.0.0.1:45886) [    ] o.a.h.h.s.d.DataNode Ending 
block pool service for: Block pool BP-1333979884-127.0.0.1-1539160532215 
(Datanode Uuid 5d0c7b98-aab8-4391-81cc-2497f6a9b8f2) service to 
serv1-vm3/127.0.0.1:45886
   [junit4]   2> 682487 WARN  
(SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[7584D7D24ABE3B7C]-worker) [    
] o.a.h.h.s.d.DirectoryScanner DirectoryScanner: shutdown has been called
   [junit4]   2> 682494 INFO  
(SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[7584D7D24ABE3B7C]-worker) [    
] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
   [junit4]   2> 682597 WARN  (DataNode: 
[[[DISK]file:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_7584D7D24ABE3B7C-003/tempDir-002/hdfsBaseDir/data/data1/,
 
[DISK]file:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_7584D7D24ABE3B7C-003/tempDir-002/hdfsBaseDir/data/data2/]]
  heartbeating to serv1-vm3/127.0.0.1:45886) [    ] 
o.a.h.h.s.d.IncrementalBlockReportManager IncrementalBlockReportManager 
interrupted
   [junit4]   2> 682598 WARN  (DataNode: 
[[[DISK]file:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_7584D7D24ABE3B7C-003/tempDir-002/hdfsBaseDir/data/data1/,
 
[DISK]file:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_7584D7D24ABE3B7C-003/tempDir-002/hdfsBaseDir/data/data2/]]
  heartbeating to serv1-vm3/127.0.0.1:45886) [    ] o.a.h.h.s.d.DataNode Ending 
block pool service for: Block pool BP-1333979884-127.0.0.1-1539160532215 
(Datanode Uuid e579b9d7-6142-491c-adb2-3a82ca1ee5ff) service to 
serv1-vm3/127.0.0.1:45886
   [junit4]   2> 682617 INFO  
(SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[7584D7D24ABE3B7C]-worker) [    
] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@serv1-vm3:0
   [junit4]   2> 682733 INFO  
(SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[7584D7D24ABE3B7C]-worker) [    
] o.a.s.c.ZkTestServer connecting to 127.0.0.1:46965 46965
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_7584D7D24ABE3B7C-003
   [junit4]   2> Oct 10, 2018 8:37:37 AM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 27 leaked 
thread(s).
   [junit4]   2> NOTE: test params are: codec=Lucene80, 
sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@59bde90b),
 locale=es-HN, timezone=Pacific/Gambier
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_172 
(64-bit)/cpus=3,threads=2,free=93850760,total=228417536
   [junit4]   2> NOTE: All tests run in this JVM: 
[AutoAddReplicasIntegrationTest, HdfsAutoAddReplicasIntegrationTest, 
AutoAddReplicasIntegrationTest, HdfsAutoAddReplicasIntegrationTest, 
HdfsAutoAddReplicasIntegrationTest]
   [junit4] Completed [10/10 (10!)] on J0 in 136.37s, 1 test, 1 failure <<< 
FAILURES!
   [junit4] 
   [junit4] 
   [junit4] Tests with failures [seed: 7584D7D24ABE3B7C]:
   [junit4]   - 
org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple
   [junit4]   - 
org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple
   [junit4]   - 
org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest.testSimple
   [junit4]   - 
org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple
   [junit4]   - 
org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple
   [junit4]   - 
org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple
   [junit4]   - 
org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest.testSimple
   [junit4]   - 
org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest.testSimple
   [junit4]   - 
org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest.testSimple
   [junit4]   - 
org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest.testSimple
   [junit4] 
   [junit4] 
   [junit4] JVM J0:     0.76 ..   693.09 =   692.33s
   [junit4] JVM J1:     0.75 ..   688.38 =   687.63s
   [junit4] Execution time total: 11 minutes 33 seconds
   [junit4] Tests summary: 10 suites, 10 tests, 10 failures

BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:1567:
 The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:1092:
 There were test failures: 10 suites, 10 tests, 10 failures [seed: 
7584D7D24ABE3B7C]

Total time: 11 minutes 35 seconds

[repro] Setting last failure code to 256

[repro] Failures:
[repro]   5/5 failed: 
org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest
[repro]   5/5 failed: 
org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest
[repro] Exiting with code 256
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Archiving artifacts
Setting 
ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting 
ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all 
warnings.
<Git Blamer> GIT_COMMIT=3629e760113d8faa4b544bffafa3e8b33d2eb404, 
workspace=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris
[WARNINGS] Computing warning deltas based on reference build #2098
Recording test results
Setting 
ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting 
ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org