Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1047/

10 tests failed.
FAILED:  
org.apache.solr.cloud.CdcrReplicationDistributedZkTest.testReplicationAfterLeaderChange

Error Message:
Timeout while trying to assert number of documents @ target_collection

Stack Trace:
java.lang.AssertionError: Timeout while trying to assert number of documents @ 
target_collection
        at 
__randomizedtesting.SeedInfo.seed([C96C46BE1BDAAA32:1B9C0A5D45750C00]:0)
        at 
org.apache.solr.cloud.BaseCdcrDistributedZkTest.assertNumDocs(BaseCdcrDistributedZkTest.java:268)
        at 
org.apache.solr.cloud.CdcrReplicationDistributedZkTest.testReplicationAfterLeaderChange(CdcrReplicationDistributedZkTest.java:291)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: expected:<10> but was:<4>
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at 
org.apache.solr.cloud.BaseCdcrDistributedZkTest.assertNumDocs(BaseCdcrDistributedZkTest.java:259)
        ... 42 more


FAILED:  org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test

Error Message:
Expected 2 of 3 replicas to be active but only found 1; 
[core_node3:{"core":"c8n_1x3_lf_shard1_replica2","base_url":"http://127.0.0.1:57483","node_name":"127.0.0.1:57483_","state":"active","leader":"true"}];
 clusterState: DocCollection(c8n_1x3_lf)={   "replicationFactor":"3",   
"shards":{"shard1":{       "range":"80000000-7fffffff",       "state":"active", 
      "replicas":{         "core_node1":{           
"core":"c8n_1x3_lf_shard1_replica3",           
"base_url":"http://127.0.0.1:57242";,           "node_name":"127.0.0.1:57242_",  
         "state":"down"},         "core_node2":{           "state":"down",      
     "base_url":"http://127.0.0.1:47440";,           
"core":"c8n_1x3_lf_shard1_replica1",           "node_name":"127.0.0.1:47440_"}, 
        "core_node3":{           "core":"c8n_1x3_lf_shard1_replica2",           
"base_url":"http://127.0.0.1:57483";,           "node_name":"127.0.0.1:57483_",  
         "state":"active",           "leader":"true"}}}},   
"router":{"name":"compositeId"},   "maxShardsPerNode":"1",   
"autoAddReplicas":"false"}

Stack Trace:
java.lang.AssertionError: Expected 2 of 3 replicas to be active but only found 
1; 
[core_node3:{"core":"c8n_1x3_lf_shard1_replica2","base_url":"http://127.0.0.1:57483","node_name":"127.0.0.1:57483_","state":"active","leader":"true"}];
 clusterState: DocCollection(c8n_1x3_lf)={
  "replicationFactor":"3",
  "shards":{"shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{
        "core_node1":{
          "core":"c8n_1x3_lf_shard1_replica3",
          "base_url":"http://127.0.0.1:57242";,
          "node_name":"127.0.0.1:57242_",
          "state":"down"},
        "core_node2":{
          "state":"down",
          "base_url":"http://127.0.0.1:47440";,
          "core":"c8n_1x3_lf_shard1_replica1",
          "node_name":"127.0.0.1:47440_"},
        "core_node3":{
          "core":"c8n_1x3_lf_shard1_replica2",
          "base_url":"http://127.0.0.1:57483";,
          "node_name":"127.0.0.1:57483_",
          "state":"active",
          "leader":"true"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false"}
        at 
__randomizedtesting.SeedInfo.seed([C96C46BE1BDAAA32:41387964B526C7CA]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at 
org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.testRf3WithLeaderFailover(LeaderFailoverAfterPartitionTest.java:170)
        at 
org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test(LeaderFailoverAfterPartitionTest.java:57)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.RestartWhileUpdatingTest.test

Error Message:
There are still nodes recoverying - waited for 320 seconds

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 320 
seconds
        at 
__randomizedtesting.SeedInfo.seed([C96C46BE1BDAAA32:41387964B526C7CA]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:182)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:862)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1418)
        at 
org.apache.solr.cloud.RestartWhileUpdatingTest.test(RestartWhileUpdatingTest.java:144)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.RestartWhileUpdatingTest

Error Message:
2 threads leaked from SUITE scope at 
org.apache.solr.cloud.RestartWhileUpdatingTest:     1) Thread[id=21814, 
name=searcherExecutor-4790-thread-1, state=WAITING, 
group=TGRP-RestartWhileUpdatingTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    2) Thread[id=21745, 
name=searcherExecutor-4768-thread-1, state=WAITING, 
group=TGRP-RestartWhileUpdatingTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE 
scope at org.apache.solr.cloud.RestartWhileUpdatingTest: 
   1) Thread[id=21814, name=searcherExecutor-4790-thread-1, state=WAITING, 
group=TGRP-RestartWhileUpdatingTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=21745, name=searcherExecutor-4768-thread-1, state=WAITING, 
group=TGRP-RestartWhileUpdatingTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        at __randomizedtesting.SeedInfo.seed([C96C46BE1BDAAA32]:0)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.RestartWhileUpdatingTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) 
Thread[id=21814, name=searcherExecutor-4790-thread-1, state=WAITING, 
group=TGRP-RestartWhileUpdatingTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    2) Thread[id=21745, 
name=searcherExecutor-4768-thread-1, state=WAITING, 
group=TGRP-RestartWhileUpdatingTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie 
threads that couldn't be terminated:
   1) Thread[id=21814, name=searcherExecutor-4790-thread-1, state=WAITING, 
group=TGRP-RestartWhileUpdatingTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=21745, name=searcherExecutor-4768-thread-1, state=WAITING, 
group=TGRP-RestartWhileUpdatingTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        at __randomizedtesting.SeedInfo.seed([C96C46BE1BDAAA32]:0)


FAILED:  org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest.test

Error Message:
shard4 is not consistent.  Got 59 from https://127.0.0.1:36349/collection1 
(previous client) and got 60 from https://127.0.0.1:53360/collection1

Stack Trace:
java.lang.AssertionError: shard4 is not consistent.  Got 59 from 
https://127.0.0.1:36349/collection1 (previous client) and got 60 from 
https://127.0.0.1:53360/collection1
        at 
__randomizedtesting.SeedInfo.seed([C96C46BE1BDAAA32:41387964B526C7CA]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1271)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1250)
        at 
org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test(ChaosMonkeySafeLeaderTest.java:162)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest

Error Message:
32 threads leaked from SUITE scope at 
org.apache.solr.cloud.CdcrReplicationDistributedZkTest:     1) Thread[id=66340, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=70863, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=62223, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=61933, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=65868, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=61932, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=62366, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=62224, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=62364, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=62367, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   11) Thread[id=62620, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   12) Thread[id=69415, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   13) Thread[id=65871, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   14) Thread[id=70865, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   15) Thread[id=61930, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   16) Thread[id=62365, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   17) Thread[id=66337, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   18) Thread[id=62221, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   19) Thread[id=69417, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   20) Thread[id=69416, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   21) Thread[id=65870, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   22) Thread[id=62618, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   23) Thread[id=61931, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   24) Thread[id=69418, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   25) Thread[id=70866, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   26) Thread[id=70864, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   27) Thread[id=65869, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   28) Thread[id=62222, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   29) Thread[id=66338, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   30) Thread[id=66339, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   31) Thread[id=62619, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   32) Thread[id=62621, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 32 threads leaked from 
SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest: 
   1) Thread[id=66340, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=70863, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=62223, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=61933, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=65868, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=61932, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=62366, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=62224, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   9) Thread[id=62364, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  10) Thread[id=62367, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  11) Thread[id=62620, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  12) Thread[id=69415, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  13) Thread[id=65871, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  14) Thread[id=70865, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  15) Thread[id=61930, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  16) Thread[id=62365, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  17) Thread[id=66337, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  18) Thread[id=62221, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  19) Thread[id=69417, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  20) Thread[id=69416, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  21) Thread[id=65870, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  22) Thread[id=62618, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  23) Thread[id=61931, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  24) Thread[id=69418, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  25) Thread[id=70866, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  26) Thread[id=70864, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  27) Thread[id=65869, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  28) Thread[id=62222, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  29) Thread[id=66338, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  30) Thread[id=66339, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  31) Thread[id=62619, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  32) Thread[id=62621, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
        at __randomizedtesting.SeedInfo.seed([C96C46BE1BDAAA32]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrRequestHandlerTest

Error Message:
ObjectTracker found 8 object(s) that were not released!!! [InternalHttpClient, 
InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, 
InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 8 object(s) that were not 
released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, 
InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, 
InternalHttpClient]
        at __randomizedtesting.SeedInfo.seed([C96C46BE1BDAAA32]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNull(Assert.java:551)
        at 
org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:256)
        at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrRequestHandlerTest

Error Message:
8 threads leaked from SUITE scope at 
org.apache.solr.cloud.CdcrRequestHandlerTest:     1) Thread[id=52541, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native 
Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=52153, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native 
Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=52540, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native 
Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=52154, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native 
Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=52543, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native 
Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=52155, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native 
Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=52156, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native 
Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=52542, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native 
Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 8 threads leaked from SUITE 
scope at org.apache.solr.cloud.CdcrRequestHandlerTest: 
   1) Thread[id=52541, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=52153, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=52540, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=52154, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=52543, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=52155, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=52156, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=52542, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
        at __randomizedtesting.SeedInfo.seed([C96C46BE1BDAAA32]:0)


FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: https://127.0.0.1:45667

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting 
response from server at: https://127.0.0.1:45667
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:617)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:400)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:516)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:180)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:170)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
        at sun.security.ssl.InputRecord.read(InputRecord.java:503)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
        at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
        at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
        at 
org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
        at 
org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
        at 
org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
        at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
        at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
        at 
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
        at 
org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
        at 
org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
        at 
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
        at 
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
        at 
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
        at 
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
        at 
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at 
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:511)
        ... 47 more




Build Log:
[...truncated 11350 lines...]
   [junit4] Suite: 
org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_C96C46BE1BDAAA32-001/init-core-data-001
   [junit4]   2> 109858 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[C96C46BE1BDAAA32]-worker) [    
] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: 
@org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 109870 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[C96C46BE1BDAAA32]-worker) [    
] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 111746 WARN  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[C96C46BE1BDAAA32]-worker) [    
] o.a.h.u.NativeCodeLoader Unable to load native-hadoop library for your 
platform... using builtin-java classes where applicable
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 113231 WARN  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[C96C46BE1BDAAA32]-worker) [    
] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried 
hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 113813 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[C96C46BE1BDAAA32]-worker) [    
] o.m.log Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via 
org.mortbay.log.Slf4jLog
   [junit4]   2> 113820 WARN  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[C96C46BE1BDAAA32]-worker) [    
] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 113983 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[C96C46BE1BDAAA32]-worker) [    
] o.m.log jetty-6.1.26
   [junit4]   2> 114046 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[C96C46BE1BDAAA32]-worker) [    
] o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/hdfs
 to ./temp/Jetty_localhost_60307_hdfs____kkoi7e/webapp
   [junit4]   2> 114513 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[C96C46BE1BDAAA32]-worker) [    
] o.m.log NO JSP Support for /, did not find 
org.apache.jasper.servlet.JspServlet
   [junit4]   2> 115456 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[C96C46BE1BDAAA32]-worker) [    
] o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:60307
   [junit4]   2> 117900 WARN  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[C96C46BE1BDAAA32]-worker) [    
] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 117905 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[C96C46BE1BDAAA32]-worker) [    
] o.m.log jetty-6.1.26
   [junit4]   2> 117919 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[C96C46BE1BDAAA32]-worker) [    
] o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode
 to ./temp/Jetty_localhost_59698_datanode____.4xz8bs/webapp
   [junit4]   2> 118055 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[C96C46BE1BDAAA32]-worker) [    
] o.m.log NO JSP Support for /, did not find 
org.apache.jasper.servlet.JspServlet
   [junit4]   2> 118361 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[C96C46BE1BDAAA32]-worker) [    
] o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:59698
   [junit4]   2> 118531 WARN  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[C96C46BE1BDAAA32]-worker) [    
] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 118533 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[C96C46BE1BDAAA32]-worker) [    
] o.m.log jetty-6.1.26
   [junit4]   2> 118547 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[C96C46BE1BDAAA32]-worker) [    
] o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode
 to ./temp/Jetty_localhost_33345_datanode____.hhb91h/webapp
   [junit4]   2> 118759 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[C96C46BE1BDAAA32]-worker) [    
] o.m.log NO JSP Support for /, did not find 
org.apache.jasper.servlet.JspServlet
   [junit4]   2> 119144 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[C96C46BE1BDAAA32]-worker) [    
] o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33345
   [junit4]   2> 121341 INFO  (IPC Server handler 6 on 44047) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-0fda6a11-f3a0-4a5f-9cfc-4fea973e2404 node DatanodeRegistration(127.0.0.1, 
datanodeUuid=d206459b-488f-40b1-a21d-fbc3e1419762, infoPort=33345, 
ipcPort=41611, storageInfo=lv=-56;cid=testClusterID;nsid=1465989094;c=0), 
blocks: 0, hasStaleStorages: true, processing time: 0 msecs
   [junit4]   2> 121341 INFO  (IPC Server handler 6 on 44047) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-f66963f1-9f6a-4911-8be6-de39a32f9942 node DatanodeRegistration(127.0.0.1, 
datanodeUuid=d206459b-488f-40b1-a21d-fbc3e1419762, infoPort=33345, 
ipcPort=41611, storageInfo=lv=-56;cid=testClusterID;nsid=1465989094;c=0), 
blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 121344 INFO  (IPC Server handler 3 on 44047) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-732c7e9d-bf4c-4123-98ac-11ce9009ddce node DatanodeRegistration(127.0.0.1, 
datanodeUuid=4ffc24d0-533c-4830-bf03-93ac3bc9c902, infoPort=59698, 
ipcPort=40599, storageInfo=lv=-56;cid=testClusterID;nsid=1465989094;c=0), 
blocks: 0, hasStaleStorages: true, processing time: 3 msecs
   [junit4]   2> 121352 INFO  (IPC Server handler 3 on 44047) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-55cbc1cf-c940-45fe-ae80-bd48d390add6 node DatanodeRegistration(127.0.0.1, 
datanodeUuid=4ffc24d0-533c-4830-bf03-93ac3bc9c902, infoPort=59698, 
ipcPort=40599, storageInfo=lv=-56;cid=testClusterID;nsid=1465989094;c=0), 
blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 122150 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 122150 INFO  (Thread-241) [    ] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 122150 INFO  (Thread-241) [    ] o.a.s.c.ZkTestServer Starting 
server
   [junit4]   2> 122381 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.ZkTestServer start zk server on port:57827
   [junit4]   2> 122381 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 122400 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 122560 INFO  (zkCallback-76-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@6f19f4c7 
name:ZooKeeperConnection Watcher:127.0.0.1:57827 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 122560 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 122560 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 122560 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 122570 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 122584 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 122676 INFO  (zkCallback-77-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@4ff47f6b 
name:ZooKeeperConnection Watcher:127.0.0.1:57827/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 122679 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 122680 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 122680 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 122685 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 122686 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 122727 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 122729 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
 to /configs/conf1/solrconfig.xml
   [junit4]   2> 122734 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 122833 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/schema.xml
 to /configs/conf1/schema.xml
   [junit4]   2> 122833 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 122840 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
 to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 122840 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.c.SolrZkClient makePath: 
/configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 122841 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
 to /configs/conf1/stopwords.txt
   [junit4]   2> 122841 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 122842 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/protwords.txt
 to /configs/conf1/protwords.txt
   [junit4]   2> 122842 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 122843 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/currency.xml
 to /configs/conf1/currency.xml
   [junit4]   2> 122843 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 122844 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
 to /configs/conf1/enumsConfig.xml
   [junit4]   2> 122844 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 122845 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
 to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 122845 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 122846 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
 to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 122846 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 122847 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
 to /configs/conf1/old_synonyms.txt
   [junit4]   2> 122847 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 122848 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
 to /configs/conf1/synonyms.txt
   [junit4]   2> 122848 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 122854 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 122871 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 122926 INFO  (zkCallback-79-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@29222b75 
name:ZooKeeperConnection Watcher:127.0.0.1:57827/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 122926 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 122927 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 123063 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.SolrTestCaseJ4 Writing core.properties file to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_C96C46BE1BDAAA32-001/control-001/cores/collection1
   [junit4]   2> 123094 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 123127 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@5788709f{/,null,AVAILABLE}
   [junit4]   2> 123357 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.e.j.s.ServerConnector Started ServerConnector@6ad101a7{SSL,[ssl, 
http/1.1]}{127.0.0.1:33488}
   [junit4]   2> 123357 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.e.j.s.Server Started @126781ms
   [junit4]   2> 123357 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=hdfs://localhost:44047/hdfs__localhost_44047__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-master_solr_build_solr-core_test_J1_temp_solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_C96C46BE1BDAAA32-001_tempDir-002_control_data,
 hostContext=/, hostPort=33488, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_C96C46BE1BDAAA32-001/control-001/cores}
   [junit4]   2> 123367 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): 
sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 123371 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_C96C46BE1BDAAA32-001/control-001'
   [junit4]   2> 123371 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 123371 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find 
system property or JNDI)
   [junit4]   2> 123371 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 123376 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 123388 INFO  (zkCallback-80-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@3068f494 
name:ZooKeeperConnection Watcher:127.0.0.1:57827/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 123388 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 123388 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 123389 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 123389 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_C96C46BE1BDAAA32-001/control-001/solr.xml
   [junit4]   2> 123394 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.CorePropertiesLocator Config-defined core root directory: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_C96C46BE1BDAAA32-001/control-001/cores
   [junit4]   2> 123394 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.CoreContainer New CoreContainer 1990332651
   [junit4]   2> 123394 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.CoreContainer Loading cores into CoreContainer 
[instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_C96C46BE1BDAAA32-001/control-001]
   [junit4]   2> 123395 WARN  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.CoreContainer Couldn't add files from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_C96C46BE1BDAAA32-001/control-001/lib
 to classpath: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_C96C46BE1BDAAA32-001/control-001/lib
   [junit4]   2> 123395 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme 
: ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 
10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 
5,sizeOfQueue : -1,fairnessPolicy : false,
   [junit4]   2> 123400 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
   [junit4]   2> 123405 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 123405 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.l.LogWatcher Registering Log Listener [Log4j 
(org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 123405 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:57827/solr
   [junit4]   2> 123432 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 123432 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 123453 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 123493 INFO  (zkCallback-83-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@2a2f08f name:ZooKeeperConnection 
Watcher:127.0.0.1:57827 got event WatchedEvent state:SyncConnected type:None 
path:null path:null type:None
   [junit4]   2> 123494 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 123495 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 123501 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) 
[n:127.0.0.1:33488_    ] o.a.s.c.ZkController Added new OnReconnect listener 
org.apache.solr.cloud.ZkController$$Lambda$36/913950147@19846a45
   [junit4]   2> 123501 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) 
[n:127.0.0.1:33488_    ] o.a.s.c.c.ConnectionManager Waiting for client to 
connect to ZooKeeper
   [junit4]   2> 123515 INFO  
(zkCallback-84-thread-1-processing-n:127.0.0.1:33488_) [n:127.0.0.1:33488_    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@43f00761 
name:ZooKeeperConnection Watcher:127.0.0.1:57827/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 123515 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) 
[n:127.0.0.1:33488_    ] o.a.s.c.c.ConnectionManager Client is connected to 
ZooKeeper
   [junit4]   2> 123516 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) 
[n:127.0.0.1:33488_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 123517 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) 
[n:127.0.0.1:33488_    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-queue-work
   [junit4]   2> 123519 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) 
[n:127.0.0.1:33488_    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-map-running
   [junit4]   2> 123520 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) 
[n:127.0.0.1:33488_    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-map-completed
   [junit4]   2> 123521 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) 
[n:127.0.0.1:33488_    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-map-failure
   [junit4]   2> 123522 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[C96C46BE1BDAAA32]) 
[n:127.0.0.1:33488_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 123527 INFO  (TEST-HdfsCollectionsAPIDistribute

[...truncated too long message...]

[junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Jun 20, 2016 10:34:22 AM 
com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=52541, name=Connection evictor, 
state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
   [junit4]   2>    2) Thread[id=52153, name=Connection evictor, 
state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
   [junit4]   2>    3) Thread[id=52540, name=Connection evictor, 
state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
   [junit4]   2>    4) Thread[id=52154, name=Connection evictor, 
state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
   [junit4]   2>    5) Thread[id=52543, name=Connection evictor, 
state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
   [junit4]   2>    6) Thread[id=52155, name=Connection evictor, 
state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
   [junit4]   2>    7) Thread[id=52156, name=Connection evictor, 
state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
   [junit4]   2>    8) Thread[id=52542, name=Connection evictor, 
state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
   [junit4]   2> Jun 20, 2016 10:34:22 AM 
com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: All leaked threads terminated.
   [junit4]   2> NOTE: test params are: 
codec=DummyCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=DUMMY,
 chunkSize=9, maxDocsPerChunk=7, blockSize=858), 
termVectorsFormat=CompressingTermVectorsFormat(compressionMode=DUMMY, 
chunkSize=9, blockSize=858)), sim=RandomSimilarity(queryNorm=true,coord=crazy): 
{}, locale=lv, timezone=AGT
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 
1.8.0_74 (64-bit)/cpus=4,threads=1,free=108009440,total=504889344
   [junit4]   2> NOTE: All tests run in this JVM: [TestLockTree, 
TestDistributedStatsComponentCardinality, 
HdfsTlogReplayBufferedWhileIndexingTest, GraphQueryTest, TestSolrQueryParser, 
TestIntervalFaceting, HdfsNNFailoverTest, SliceStateTest, RulesTest, 
ScriptEngineTest, TestGroupingSearch, TestWriterPerf, DateRangeFieldTest, 
CoreAdminHandlerTest, CleanupOldIndexTest, ShardSplitTest, 
SegmentsInfoRequestHandlerTest, DistribDocExpirationUpdateProcessorTest, 
TestLegacyFieldCache, DirectUpdateHandlerTest, BitVectorTest, 
TestRawTransformer, ZkStateWriterTest, DateMathParserTest, 
TestCSVResponseWriter, HLLSerializationTest, TestDynamicFieldResource, 
LukeRequestHandlerTest, TestCustomSort, CachingDirectoryFactoryTest, 
RollingRestartTest, TestConfigReload, DirectSolrConnectionTest, 
TestRandomRequestDistribution, BasicDistributedZk2Test, 
TestClassicSimilarityFactory, OverseerStatusTest, 
SolrCoreCheckLockOnStartupTest, UnloadDistributedZkTest, TestRestManager, 
TestDynamicLoading, TestExtendedDismaxParser, BadComponentTest, 
TestAuthorizationFramework, TestJmxMonitoredMap, TestAuthenticationFramework, 
LeaderFailoverAfterPartitionTest, BadCopyFieldTest, 
TestExclusionRuleCollectionAccess, OverriddenZkACLAndCredentialsProvidersTest, 
FieldMutatingUpdateProcessorTest, MigrateRouteKeyTest, TestInitParams, 
PluginInfoTest, OverseerTest, CollectionStateFormat2Test, TestSolrXml, 
TestCloudInspectUtil, BadIndexSchemaTest, NoCacheHeaderTest, RecoveryZkTest, 
ParsingFieldUpdateProcessorsTest, SpatialHeatmapFacetsTest, TestSchemaResource, 
TestCoreDiscovery, HighlighterConfigTest, HardAutoCommitTest, TestRecovery, 
SuggesterTest, TestConfigSetImmutable, TestIndexSearcher, 
TestSuggestSpellingConverter, TestSerializedLuceneMatchVersion, 
TestStressReorder, TestCryptoKeys, TestUseDocValuesAsStored, TestDistribIDF, 
CoreMergeIndexesAdminHandlerTest, TestShortCircuitedRequests, TestFastWriter, 
TestFreeTextSuggestions, TestSolrJ, StatelessScriptUpdateProcessorFactoryTest, 
TestTolerantUpdateProcessorRandomCloud, HdfsChaosMonkeyNothingIsSafeTest, 
TestSmileRequest, RequestHandlersTest, SimpleCollectionCreateDeleteTest, 
TestUninvertingReader, RankQueryTest, FullSolrCloudDistribCmdsTest, 
BinaryUpdateRequestHandlerTest, TestManagedSchema, CurrencyFieldXmlFileTest, 
TestTolerantSearch, DirectUpdateHandlerOptimizeTest, 
TestSubQueryTransformerDistrib, CdcrRequestHandlerTest]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 
'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=CdcrRequestHandlerTest -Dtests.seed=C96C46BE1BDAAA32 
-Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true 
-Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt 
-Dtests.locale=lv -Dtests.timezone=AGT -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J2 | CdcrRequestHandlerTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ObjectTracker found 8 
object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, 
InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, 
InternalHttpClient, InternalHttpClient]
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([C96C46BE1BDAAA32]:0)
   [junit4]    >        at 
org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:256)
   [junit4]    >        at java.lang.Thread.run(Thread.java:745)Throwable #2: 
com.carrotsearch.randomizedtesting.ThreadLeakError: 8 threads leaked from SUITE 
scope at org.apache.solr.cloud.CdcrRequestHandlerTest: 
   [junit4]    >    1) Thread[id=52541, name=Connection evictor, 
state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    2) Thread[id=52153, name=Connection evictor, 
state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    3) Thread[id=52540, name=Connection evictor, 
state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    4) Thread[id=52154, name=Connection evictor, 
state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    5) Thread[id=52543, name=Connection evictor, 
state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    6) Thread[id=52155, name=Connection evictor, 
state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    7) Thread[id=52156, name=Connection evictor, 
state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    8) Thread[id=52542, name=Connection evictor, 
state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([C96C46BE1BDAAA32]:0)
   [junit4] Completed [614/614 (6!)] on J2 in 134.68s, 3 tests, 1 failure, 1 
error <<< FAILURES!

[...truncated 18 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:747:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:684:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:59:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build.xml:233:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/common-build.xml:530:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/common-build.xml:1427:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/common-build.xml:984:
 There were test failures: 614 suites (10 ignored), 2584 tests, 5 suite-level 
errors, 1 error, 4 failures, 1278 ignored (12 assumptions) [seed: 
C96C46BE1BDAAA32]

Total time: 289 minutes 39 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to