Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/56/

9 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest.test

Error Message:
Could not load collection from ZK: collection1

Stack Trace:
org.apache.solr.common.SolrException: Could not load collection from ZK: 
collection1
        at 
__randomizedtesting.SeedInfo.seed([93F883ACB19AA413:1BACBC761F66C9EB]:0)
        at 
org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1115)
        at 
org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:648)
        at 
org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(ClusterState.java:128)
        at 
org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:108)
        at 
org.apache.solr.cloud.ChaosMonkey.logCollectionStateSummary(ChaosMonkey.java:709)
        at org.apache.solr.cloud.ChaosMonkey.wait(ChaosMonkey.java:703)
        at 
org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest.test(ChaosMonkeyNothingIsSafeWithPullReplicasTest.java:220)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: 
KeeperErrorCode = Session expired for /collections/collection1/state.json
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
        at 
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:357)
        at 
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
        at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at 
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354)
        at 
org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1127)
        at 
org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1113)
        ... 47 more


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest

Error Message:
12 threads leaked from SUITE scope at 
org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest:     1) 
Thread[id=6221, name=FullThrottleStopableIndexingThread, state=RUNNABLE, 
group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at 
java.util.WeakHashMap.get(WeakHashMap.java:403)         at 
com.carrotsearch.randomizedtesting.RandomizedContext.getPerThread(RandomizedContext.java:319)
         at 
com.carrotsearch.randomizedtesting.RandomizedContext.getRandomness(RandomizedContext.java:92)
         at 
com.carrotsearch.randomizedtesting.RandomizedContext.getRandom(RandomizedContext.java:114)
         at 
org.apache.lucene.util.LuceneTestCase.random(LuceneTestCase.java:746)         
at 
org.apache.solr.cloud.FullThrottleStoppableIndexingThread.run(FullThrottleStoppableIndexingThread.java:76)
    2) Thread[id=6210, name=StoppableSearchThread, state=BLOCKED, 
group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at 
org.apache.solr.client.solrj.impl.CloudSolrClient.getDocCollection(CloudSolrClient.java:1217)
         at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:827)
         at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
         at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)         
at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)         
at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:957)         
at 
org.apache.solr.cloud.StoppableSearchThread.run(StoppableSearchThread.java:54)  
  3) Thread[id=6331, name=zkCallback-875-thread-3-SendThread(127.0.0.1:53172), 
state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]   
      at java.lang.Thread.sleep(Native Method)         at 
org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
         at 
org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)    
     at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)    
4) Thread[id=6222, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=6455, 
name=StoppableCommitThread-SendThread(127.0.0.1:53172), state=TIMED_WAITING, 
group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)    6) 
Thread[id=6215, name=StoppableCommitThread, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:111)   
      at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70)
         at 
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354)        
 at 
org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1127)
         at 
org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1113)
         at 
org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:648)
         at 
org.apache.solr.client.solrj.impl.CloudSolrClient.getDocCollection(CloudSolrClient.java:1227)
         at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:827)
         at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
         at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)         
at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:524)         
at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:544)         
at 
org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:49)  
  7) Thread[id=6207, name=StoppableIndexingThread, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
    8) Thread[id=6332, name=zkCallback-875-thread-3-EventThread, state=WAITING, 
group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)    
9) Thread[id=6208, name=StoppableIndexingThread, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   10) Thread[id=5894, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:748)   11) Thread[id=6209, 
name=StoppableIndexingThread, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   12) Thread[id=6456, name=StoppableCommitThread-EventThread, state=WAITING, 
group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 12 threads leaked from 
SUITE scope at 
org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest: 
   1) Thread[id=6221, name=FullThrottleStopableIndexingThread, state=RUNNABLE, 
group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.util.WeakHashMap.get(WeakHashMap.java:403)
        at 
com.carrotsearch.randomizedtesting.RandomizedContext.getPerThread(RandomizedContext.java:319)
        at 
com.carrotsearch.randomizedtesting.RandomizedContext.getRandomness(RandomizedContext.java:92)
        at 
com.carrotsearch.randomizedtesting.RandomizedContext.getRandom(RandomizedContext.java:114)
        at org.apache.lucene.util.LuceneTestCase.random(LuceneTestCase.java:746)
        at 
org.apache.solr.cloud.FullThrottleStoppableIndexingThread.run(FullThrottleStoppableIndexingThread.java:76)
   2) Thread[id=6210, name=StoppableSearchThread, state=BLOCKED, 
group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.getDocCollection(CloudSolrClient.java:1217)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:827)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:957)
        at 
org.apache.solr.cloud.StoppableSearchThread.run(StoppableSearchThread.java:54)
   3) Thread[id=6331, name=zkCallback-875-thread-3-SendThread(127.0.0.1:53172), 
state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at 
org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
   4) Thread[id=6222, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=6455, name=StoppableCommitThread-SendThread(127.0.0.1:53172), 
state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)
   6) Thread[id=6215, name=StoppableCommitThread, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:111)
        at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70)
        at 
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354)
        at 
org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1127)
        at 
org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1113)
        at 
org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:648)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.getDocCollection(CloudSolrClient.java:1227)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:827)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:524)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:544)
        at 
org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:49)
   7) Thread[id=6207, name=StoppableIndexingThread, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   8) Thread[id=6332, name=zkCallback-875-thread-3-EventThread, state=WAITING, 
group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
   9) Thread[id=6208, name=StoppableIndexingThread, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
  10) Thread[id=5894, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
  11) Thread[id=6209, name=StoppableIndexingThread, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
  12) Thread[id=6456, name=StoppableCommitThread-EventThread, state=WAITING, 
group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
        at __randomizedtesting.SeedInfo.seed([93F883ACB19AA413]:0)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) 
Thread[id=6221, name=FullThrottleStopableIndexingThread, state=RUNNABLE, 
group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at 
java.lang.Object.hashCode(Native Method)         at 
java.util.WeakHashMap.hash(WeakHashMap.java:298)         at 
java.util.WeakHashMap.getEntry(WeakHashMap.java:426)         at 
java.util.WeakHashMap.containsKey(WeakHashMap.java:417)         at 
com.carrotsearch.randomizedtesting.RandomizedContext.context(RandomizedContext.java:258)
         at 
com.carrotsearch.randomizedtesting.RandomizedContext.current(RandomizedContext.java:134)
         at 
org.apache.lucene.util.LuceneTestCase.random(LuceneTestCase.java:746)         
at 
org.apache.solr.cloud.FullThrottleStoppableIndexingThread.run(FullThrottleStoppableIndexingThread.java:76)
    2) Thread[id=6210, name=StoppableSearchThread, state=RUNNABLE, 
group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at 
java.io.PrintStream.write(PrintStream.java:523)         at 
java.io.PrintStream.print(PrintStream.java:669)         at 
java.io.PrintStream.println(PrintStream.java:806)         at 
org.apache.solr.cloud.StoppableSearchThread.run(StoppableSearchThread.java:56)  
  3) Thread[id=6331, name=zkCallback-875-thread-3-SendThread(127.0.0.1:53172), 
state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]   
      at java.lang.Thread.sleep(Native Method)         at 
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)    4) 
Thread[id=6207, name=StoppableIndexingThread, state=BLOCKED, 
group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at 
java.io.PrintStream.println(PrintStream.java:805)         at 
org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:120)
    5) Thread[id=6208, name=StoppableIndexingThread, state=BLOCKED, 
group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at 
java.io.PrintStream.println(PrintStream.java:805)         at 
org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:120)
    6) Thread[id=6455, name=StoppableCommitThread-SendThread(127.0.0.1:53172), 
state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]   
      at java.lang.Thread.sleep(Native Method)         at 
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)    7) 
Thread[id=6209, name=StoppableIndexingThread, state=BLOCKED, 
group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at 
java.io.PrintStream.println(PrintStream.java:805)         at 
org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:120)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie 
threads that couldn't be terminated:
   1) Thread[id=6221, name=FullThrottleStopableIndexingThread, state=RUNNABLE, 
group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Object.hashCode(Native Method)
        at java.util.WeakHashMap.hash(WeakHashMap.java:298)
        at java.util.WeakHashMap.getEntry(WeakHashMap.java:426)
        at java.util.WeakHashMap.containsKey(WeakHashMap.java:417)
        at 
com.carrotsearch.randomizedtesting.RandomizedContext.context(RandomizedContext.java:258)
        at 
com.carrotsearch.randomizedtesting.RandomizedContext.current(RandomizedContext.java:134)
        at org.apache.lucene.util.LuceneTestCase.random(LuceneTestCase.java:746)
        at 
org.apache.solr.cloud.FullThrottleStoppableIndexingThread.run(FullThrottleStoppableIndexingThread.java:76)
   2) Thread[id=6210, name=StoppableSearchThread, state=RUNNABLE, 
group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.io.PrintStream.write(PrintStream.java:523)
        at java.io.PrintStream.print(PrintStream.java:669)
        at java.io.PrintStream.println(PrintStream.java:806)
        at 
org.apache.solr.cloud.StoppableSearchThread.run(StoppableSearchThread.java:56)
   3) Thread[id=6331, name=zkCallback-875-thread-3-SendThread(127.0.0.1:53172), 
state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)
   4) Thread[id=6207, name=StoppableIndexingThread, state=BLOCKED, 
group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.io.PrintStream.println(PrintStream.java:805)
        at 
org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:120)
   5) Thread[id=6208, name=StoppableIndexingThread, state=BLOCKED, 
group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.io.PrintStream.println(PrintStream.java:805)
        at 
org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:120)
   6) Thread[id=6455, name=StoppableCommitThread-SendThread(127.0.0.1:53172), 
state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)
   7) Thread[id=6209, name=StoppableIndexingThread, state=BLOCKED, 
group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.io.PrintStream.println(PrintStream.java:805)
        at 
org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:120)
        at __randomizedtesting.SeedInfo.seed([93F883ACB19AA413]:0)


FAILED:  
org.apache.solr.cloud.TestCloudRecovery.leaderRecoverFromLogOnStartupTest

Error Message:
Could not load collection from ZK: collection1

Stack Trace:
org.apache.solr.common.SolrException: Could not load collection from ZK: 
collection1
        at 
__randomizedtesting.SeedInfo.seed([93F883ACB19AA413:E70862E9F5BEE39C]:0)
        at 
org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1115)
        at 
org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:648)
        at 
org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(ClusterState.java:128)
        at 
org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:108)
        at 
org.apache.solr.common.cloud.ClusterStateUtil.waitForAllActiveAndLiveReplicas(ClusterStateUtil.java:70)
        at 
org.apache.solr.cloud.TestCloudRecovery.leaderRecoverFromLogOnStartupTest(TestCloudRecovery.java:99)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: 
KeeperErrorCode = Session expired for /collections/collection1/state.json
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
        at 
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:357)
        at 
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
        at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at 
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354)
        at 
org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1127)
        at 
org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1113)
        ... 44 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.UnloadDistributedZkTest

Error Message:
ObjectTracker found 1 object(s) that were not released!!! 
[MDCAwareThreadPoolExecutor] 
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor  at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:872)  at 
org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)  at 
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1024)
  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:938)  at 
org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)
  at 
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)
  at 
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)
  at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
  at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:745)  
at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:726)  
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:507)  at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
  at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
  at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
  at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
  at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) 
 at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
  at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)  
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
  at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
  at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  
at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)  
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) 
 at org.eclipse.jetty.server.Server.handle(Server.java:534)  at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)  at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)  at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)  at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) 
 at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
  at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
  at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
  at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
  at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) 
 at java.lang.Thread.run(Thread.java:748)  

Stack Trace:
java.lang.AssertionError: ObjectTracker found 1 object(s) that were not 
released!!! [MDCAwareThreadPoolExecutor]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor
        at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:872)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)
        at 
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1024)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:938)
        at 
org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)
        at 
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)
        at 
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)
        at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
        at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
        at 
org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:745)
        at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:726)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:507)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
        at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
        at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
        at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)


        at __randomizedtesting.SeedInfo.seed([93F883ACB19AA413]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNull(Assert.java:551)
        at 
org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:294)
        at sun.reflect.GeneratedMethodAccessor108.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:897)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload

Error Message:
expected:<[{indexVersion=1507121150531,generation=2,filelist=[_10v.cfe, 
_10v.cfs, _10v.si, _17e.cfe, _17e.cfs, _17e.si, _1k4.cfe, _1k4.cfs, _1k4.si, 
_1ka.cfe, _1ka.cfs, _1ka.si, _1qn.cfe, _1qn.cfs, _1qn.si, _1x0.cfe, _1x0.cfs, 
_1x0.si, _23j.cfe, _23j.cfs, _23j.si, _24e.cfe, _24e.cfs, _24e.si, _24f.fdt, 
_24f.fdx, _24f.fnm, _24f.nvd, _24f.nvm, _24f.si, _24f_Asserting_0.doc, 
_24f_Asserting_0.tim, _24f_Asserting_0.tip, _24g.fdt, _24g.fdx, _24g.fnm, 
_24g.nvd, _24g.nvm, _24g.si, _24g_Asserting_0.doc, _24g_Asserting_0.tim, 
_24g_Asserting_0.tip, _24h.fdt, _24h.fdx, _24h.fnm, _24h.nvd, _24h.nvm, 
_24h.si, _24h_Asserting_0.doc, _24h_Asserting_0.tim, _24h_Asserting_0.tip, 
_24i.fdt, _24i.fdx, _24i.fnm, _24i.nvd, _24i.nvm, _24i.si, 
_24i_Asserting_0.doc, _24i_Asserting_0.tim, _24i_Asserting_0.tip, _24j.fdt, 
_24j.fdx, _24j.fnm, _24j.nvd, _24j.nvm, _24j.si, _24j_Asserting_0.doc, 
_24j_Asserting_0.tim, _24j_Asserting_0.tip, _ui.cfe, _ui.cfs, _ui.si, 
segments_2]}]> but 
was:<[{indexVersion=1507121150531,generation=2,filelist=[_10v.cfe, _10v.cfs, 
_10v.si, _17e.cfe, _17e.cfs, _17e.si, _1k4.cfe, _1k4.cfs, _1k4.si, _1ka.cfe, 
_1ka.cfs, _1ka.si, _1qn.cfe, _1qn.cfs, _1qn.si, _1x0.cfe, _1x0.cfs, _1x0.si, 
_23j.cfe, _23j.cfs, _23j.si, _24e.cfe, _24e.cfs, _24e.si, _24f.fdt, _24f.fdx, 
_24f.fnm, _24f.nvd, _24f.nvm, _24f.si, _24f_Asserting_0.doc, 
_24f_Asserting_0.tim, _24f_Asserting_0.tip, _24g.fdt, _24g.fdx, _24g.fnm, 
_24g.nvd, _24g.nvm, _24g.si, _24g_Asserting_0.doc, _24g_Asserting_0.tim, 
_24g_Asserting_0.tip, _24h.fdt, _24h.fdx, _24h.fnm, _24h.nvd, _24h.nvm, 
_24h.si, _24h_Asserting_0.doc, _24h_Asserting_0.tim, _24h_Asserting_0.tip, 
_24i.fdt, _24i.fdx, _24i.fnm, _24i.nvd, _24i.nvm, _24i.si, 
_24i_Asserting_0.doc, _24i_Asserting_0.tim, _24i_Asserting_0.tip, _24j.fdt, 
_24j.fdx, _24j.fnm, _24j.nvd, _24j.nvm, _24j.si, _24j_Asserting_0.doc, 
_24j_Asserting_0.tim, _24j_Asserting_0.tip, _ui.cfe, _ui.cfs, _ui.si, 
segments_2]}, {indexVersion=1507121150531,generation=3,filelist=[_10v.cfe, 
_10v.cfs, _10v.si, _17e.cfe, _17e.cfs, _17e.si, _1k4.cfe, _1k4.cfs, _1k4.si, 
_1ka.cfe, _1ka.cfs, _1ka.si, _1qn.cfe, _1qn.cfs, _1qn.si, _1x0.cfe, _1x0.cfs, 
_1x0.si, _23j.cfe, _23j.cfs, _23j.si, _24k.cfe, _24k.cfs, _24k.si, _ui.cfe, 
_ui.cfs, _ui.si, segments_3]}]>

Stack Trace:
java.lang.AssertionError: 
expected:<[{indexVersion=1507121150531,generation=2,filelist=[_10v.cfe, 
_10v.cfs, _10v.si, _17e.cfe, _17e.cfs, _17e.si, _1k4.cfe, _1k4.cfs, _1k4.si, 
_1ka.cfe, _1ka.cfs, _1ka.si, _1qn.cfe, _1qn.cfs, _1qn.si, _1x0.cfe, _1x0.cfs, 
_1x0.si, _23j.cfe, _23j.cfs, _23j.si, _24e.cfe, _24e.cfs, _24e.si, _24f.fdt, 
_24f.fdx, _24f.fnm, _24f.nvd, _24f.nvm, _24f.si, _24f_Asserting_0.doc, 
_24f_Asserting_0.tim, _24f_Asserting_0.tip, _24g.fdt, _24g.fdx, _24g.fnm, 
_24g.nvd, _24g.nvm, _24g.si, _24g_Asserting_0.doc, _24g_Asserting_0.tim, 
_24g_Asserting_0.tip, _24h.fdt, _24h.fdx, _24h.fnm, _24h.nvd, _24h.nvm, 
_24h.si, _24h_Asserting_0.doc, _24h_Asserting_0.tim, _24h_Asserting_0.tip, 
_24i.fdt, _24i.fdx, _24i.fnm, _24i.nvd, _24i.nvm, _24i.si, 
_24i_Asserting_0.doc, _24i_Asserting_0.tim, _24i_Asserting_0.tip, _24j.fdt, 
_24j.fdx, _24j.fnm, _24j.nvd, _24j.nvm, _24j.si, _24j_Asserting_0.doc, 
_24j_Asserting_0.tim, _24j_Asserting_0.tip, _ui.cfe, _ui.cfs, _ui.si, 
segments_2]}]> but 
was:<[{indexVersion=1507121150531,generation=2,filelist=[_10v.cfe, _10v.cfs, 
_10v.si, _17e.cfe, _17e.cfs, _17e.si, _1k4.cfe, _1k4.cfs, _1k4.si, _1ka.cfe, 
_1ka.cfs, _1ka.si, _1qn.cfe, _1qn.cfs, _1qn.si, _1x0.cfe, _1x0.cfs, _1x0.si, 
_23j.cfe, _23j.cfs, _23j.si, _24e.cfe, _24e.cfs, _24e.si, _24f.fdt, _24f.fdx, 
_24f.fnm, _24f.nvd, _24f.nvm, _24f.si, _24f_Asserting_0.doc, 
_24f_Asserting_0.tim, _24f_Asserting_0.tip, _24g.fdt, _24g.fdx, _24g.fnm, 
_24g.nvd, _24g.nvm, _24g.si, _24g_Asserting_0.doc, _24g_Asserting_0.tim, 
_24g_Asserting_0.tip, _24h.fdt, _24h.fdx, _24h.fnm, _24h.nvd, _24h.nvm, 
_24h.si, _24h_Asserting_0.doc, _24h_Asserting_0.tim, _24h_Asserting_0.tip, 
_24i.fdt, _24i.fdx, _24i.fnm, _24i.nvd, _24i.nvm, _24i.si, 
_24i_Asserting_0.doc, _24i_Asserting_0.tim, _24i_Asserting_0.tip, _24j.fdt, 
_24j.fdx, _24j.fnm, _24j.nvd, _24j.nvm, _24j.si, _24j_Asserting_0.doc, 
_24j_Asserting_0.tim, _24j_Asserting_0.tip, _ui.cfe, _ui.cfs, _ui.si, 
segments_2]}, {indexVersion=1507121150531,generation=3,filelist=[_10v.cfe, 
_10v.cfs, _10v.si, _17e.cfe, _17e.cfs, _17e.si, _1k4.cfe, _1k4.cfs, _1k4.si, 
_1ka.cfe, _1ka.cfs, _1ka.si, _1qn.cfe, _1qn.cfs, _1qn.si, _1x0.cfe, _1x0.cfs, 
_1x0.si, _23j.cfe, _23j.cfs, _23j.si, _24k.cfe, _24k.cfs, _24k.si, _ui.cfe, 
_ui.cfs, _ui.si, segments_3]}]>
        at 
__randomizedtesting.SeedInfo.seed([93F883ACB19AA413:B62F989CC1D2AA10]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:147)
        at 
org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload(TestReplicationHandler.java:1277)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.UnloadDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=5627, 
name=testExecutor-1980-thread-11, state=RUNNABLE, 
group=TGRP-UnloadDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=5627, name=testExecutor-1980-thread-11, 
state=RUNNABLE, group=TGRP-UnloadDistributedZkTest]
        at 
__randomizedtesting.SeedInfo.seed([93F883ACB19AA413:1BACBC761F66C9EB]:0)
Caused by: java.lang.RuntimeException: 
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting 
response from server at: http://127.0.0.1:52851
        at __randomizedtesting.SeedInfo.seed([93F883ACB19AA413]:0)
        at 
org.apache.solr.cloud.BasicDistributedZkTest.lambda$createCollectionInOneInstance$0(BasicDistributedZkTest.java:595)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout occured 
while waiting response from server at: http://127.0.0.1:52851
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:638)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
        at 
org.apache.solr.cloud.BasicDistributedZkTest.lambda$createCollectionInOneInstance$0(BasicDistributedZkTest.java:593)
        ... 4 more
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at 
org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at 
org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at 
org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at 
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at 
org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at 
org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at 
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at 
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at 
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at 
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at 
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at 
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:526)
        ... 9 more


FAILED:  org.apache.solr.cloud.hdfs.HdfsUnloadDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=13589, 
name=testExecutor-2279-thread-17, state=RUNNABLE, 
group=TGRP-HdfsUnloadDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=13589, name=testExecutor-2279-thread-17, 
state=RUNNABLE, group=TGRP-HdfsUnloadDistributedZkTest]
Caused by: java.lang.RuntimeException: 
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting 
response from server at: http://127.0.0.1:33552
        at __randomizedtesting.SeedInfo.seed([93F883ACB19AA413]:0)
        at 
org.apache.solr.cloud.BasicDistributedZkTest.lambda$createCollectionInOneInstance$0(BasicDistributedZkTest.java:595)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout occured 
while waiting response from server at: http://127.0.0.1:33552
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:638)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
        at 
org.apache.solr.cloud.BasicDistributedZkTest.lambda$createCollectionInOneInstance$0(BasicDistributedZkTest.java:593)
        ... 4 more
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at 
org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at 
org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at 
org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at 
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at 
org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at 
org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at 
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at 
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at 
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at 
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at 
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at 
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:526)
        ... 9 more


FAILED:  org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test

Error Message:
expected:<2> but was:<4>

Stack Trace:
java.lang.AssertionError: expected:<2> but was:<4>
        at 
__randomizedtesting.SeedInfo.seed([93F883ACB19AA413:1BACBC761F66C9EB]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at 
org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.assertSliceAndReplicaCount(SharedFSAutoReplicaFailoverTest.java:403)
        at 
org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.testBasics(SharedFSAutoReplicaFailoverTest.java:248)
        at 
org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test(SharedFSAutoReplicaFailoverTest.java:136)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 12966 lines...]
   [junit4] Suite: org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest
   [junit4]   2> 638732 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[93F883ACB19AA413]-worker) [    ] 
o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: 
test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_93F883ACB19AA413-001/init-core-data-001
   [junit4]   2> 638735 WARN  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[93F883ACB19AA413]-worker) [    ] 
o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=267 numCloses=267
   [junit4]   2> 638735 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[93F883ACB19AA413]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) 
w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 638777 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[93F883ACB19AA413]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: 
@org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=None)
   [junit4]   2> 638777 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[93F883ACB19AA413]-worker) [    ] 
o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 648201 WARN  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[93F883ACB19AA413]-worker) [    ] 
o.a.h.u.NativeCodeLoader Unable to load native-hadoop library for your 
platform... using builtin-java classes where applicable
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 655436 WARN  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[93F883ACB19AA413]-worker) [    ] 
o.a.h.m.i.MetricsConfig Cannot locate configuration: tried 
hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 656997 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[93F883ACB19AA413]-worker) [    ] 
o.m.log Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via 
org.mortbay.log.Slf4jLog
   [junit4]   2> 657141 WARN  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[93F883ACB19AA413]-worker) [    ] 
o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 660120 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[93F883ACB19AA413]-worker) [    ] 
o.m.log jetty-6.1.26
   [junit4]   2> 660509 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[93F883ACB19AA413]-worker) [    ] 
o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/hdfs
 to ./temp/Jetty_localhost_45416_hdfs____.x5jnku/webapp
   [junit4]   2> 664862 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[93F883ACB19AA413]-worker) [    ] 
o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45416
   [junit4]   2> 682703 WARN  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[93F883ACB19AA413]-worker) [    ] 
o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 682706 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[93F883ACB19AA413]-worker) [    ] 
o.m.log jetty-6.1.26
   [junit4]   2> 684046 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[93F883ACB19AA413]-worker) [    ] 
o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode
 to ./temp/Jetty_localhost_43750_datanode____.lj5mcm/webapp
   [junit4]   2> 689857 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[93F883ACB19AA413]-worker) [    ] 
o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43750
   [junit4]   2> 694104 WARN  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[93F883ACB19AA413]-worker) [    ] 
o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 694116 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[93F883ACB19AA413]-worker) [    ] 
o.m.log jetty-6.1.26
   [junit4]   2> 694277 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[93F883ACB19AA413]-worker) [    ] 
o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode
 to ./temp/Jetty_localhost_40125_datanode____.2wtyap/webapp
   [junit4]   2> 701754 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[93F883ACB19AA413]-worker) [    ] 
o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40125
   [junit4]   2> 710765 ERROR (DataNode: 
[[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_93F883ACB19AA413-001/tempDir-001/hdfsBaseDir/data/data1/,
 
[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_93F883ACB19AA413-001/tempDir-001/hdfsBaseDir/data/data2/]]
  heartbeating to localhost/127.0.0.1:37663) [    ] 
o.a.h.h.s.d.DirectoryScanner 
dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 
ms/sec. Assuming default value of 1000
   [junit4]   2> 711129 ERROR (DataNode: 
[[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_93F883ACB19AA413-001/tempDir-001/hdfsBaseDir/data/data3/,
 
[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_93F883ACB19AA413-001/tempDir-001/hdfsBaseDir/data/data4/]]
  heartbeating to localhost/127.0.0.1:37663) [    ] 
o.a.h.h.s.d.DirectoryScanner 
dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 
ms/sec. Assuming default value of 1000
   [junit4]   2> 715343 INFO  (Block report processor) [    ] BlockStateChange 
BLOCK* processReport 0x1a2f461e809dc3: from storage 
DS-6a655397-f967-4ddb-8113-40d611b6dddb node 
DatanodeRegistration(127.0.0.1:58442, 
datanodeUuid=21a56a47-4ffc-4762-a538-df52f644cdc7, infoPort=49125, 
infoSecurePort=0, ipcPort=53774, 
storageInfo=lv=-56;cid=testClusterID;nsid=818447236;c=0), blocks: 0, 
hasStaleStorage: true, processing time: 70 msecs
   [junit4]   2> 715361 INFO  (Block report processor) [    ] BlockStateChange 
BLOCK* processReport 0x1a2f461db5c28e: from storage 
DS-bab6ae00-2ba5-4efb-9333-2d403c9fc9d9 node 
DatanodeRegistration(127.0.0.1:59042, 
datanodeUuid=e6596a48-1289-40aa-90bb-54cd35511ff3, infoPort=52725, 
infoSecurePort=0, ipcPort=53689, 
storageInfo=lv=-56;cid=testClusterID;nsid=818447236;c=0), blocks: 0, 
hasStaleStorage: true, processing time: 18 msecs
   [junit4]   2> 715361 INFO  (Block report processor) [    ] BlockStateChange 
BLOCK* processReport 0x1a2f461e809dc3: from storage 
DS-60be838a-2b21-47f0-b0e6-c158f396eea8 node 
DatanodeRegistration(127.0.0.1:58442, 
datanodeUuid=21a56a47-4ffc-4762-a538-df52f644cdc7, infoPort=49125, 
infoSecurePort=0, ipcPort=53774, 
storageInfo=lv=-56;cid=testClusterID;nsid=818447236;c=0), blocks: 0, 
hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 715380 INFO  (Block report processor) [    ] BlockStateChange 
BLOCK* processReport 0x1a2f461db5c28e: from storage 
DS-b550d8f6-9c95-4aa3-ab21-6b0bcf67387b node 
DatanodeRegistration(127.0.0.1:59042, 
datanodeUuid=e6596a48-1289-40aa-90bb-54cd35511ff3, infoPort=52725, 
infoSecurePort=0, ipcPort=53689, 
storageInfo=lv=-56;cid=testClusterID;nsid=818447236;c=0), blocks: 0, 
hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 719515 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[93F883ACB19AA413]) [    ] 
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 719516 INFO  (Thread-412) [    ] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 719516 INFO  (Thread-412) [    ] o.a.s.c.ZkTestServer Starting 
server
   [junit4]   2> 719716 ERROR (Thread-412) [    ] o.a.z.s.ZooKeeperServer 
ZKShutdownHandler is not registered, so ZooKeeper server won't take any action 
on ERROR or SHUTDOWN server state changes
   [junit4]   2> 719743 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[93F883ACB19AA413]) [    ] 
o.a.s.c.ZkTestServer start zk server on port:51856
   [junit4]   2> 719813 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from 
client sessionid 0x15ee741d7520000, likely client has closed socket
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 720042 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[93F883ACB19AA413]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
 to /configs/conf1/solrconfig.xml
   [junit4]   2> 720046 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[93F883ACB19AA413]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml
 to /configs/conf1/schema.xml
   [junit4]   2> 720061 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[93F883ACB19AA413]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
 to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 720069 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[93F883ACB19AA413]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
 to /configs/conf1/stopwords.txt
   [junit4]   2> 720069 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[93F883ACB19AA413]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt
 to /configs/conf1/protwords.txt
   [junit4]   2> 720081 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[93F883ACB19AA413]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml
 to /configs/conf1/currency.xml
   [junit4]   2> 720089 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[93F883ACB19AA413]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
 to /configs/conf1/enumsConfig.xml
   [junit4]   2> 720119 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[93F883ACB19AA413]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
 to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 720125 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[93F883ACB19AA413]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
 to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 720133 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[93F883ACB19AA413]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
 to /configs/conf1/old_synonyms.txt
   [junit4]   2> 720141 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[93F883ACB19AA413]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
 to /configs/conf1/synonyms.txt
   [junit4]   2> 720165 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[93F883ACB19AA413]) [    ] 
o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly 
asked otherwise
   [junit4]   2> 721674 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[93F883ACB19AA413]) [    ] 
o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 721682 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[93F883ACB19AA413]) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@1aca5b1{/,null,AVAILABLE}
   [junit4]   2> 721682 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[93F883ACB19AA413]) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@5ab6f743{HTTP/1.1,[http/1.1]}{127.0.0.1:35965}
   [junit4]   2> 721682 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[93F883ACB19AA413]) [    ] 
o.e.j.s.Server Started @749725ms
   [junit4]   2> 721682 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[93F883ACB19AA413]) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/, hostPort=35965, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_93F883ACB19AA413-001/control-001/cores}
   [junit4]   2> 721682 ERROR 
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[93F883ACB19AA413]) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 721682 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[93F883ACB19AA413]) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 
7.1.0
   [junit4]   2> 721682 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[93F883ACB19AA413]) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 721683 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[93F883ACB19AA413]) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config 
dir: null
   [junit4]   2> 721683 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[93F883ACB19AA413]) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2017-10-04T12:00:07.426Z
   [junit4]   2> 721897 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[93F883ACB19AA413]) [    ] 
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 721897 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[93F883ACB19AA413]) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_93F883ACB19AA413-001/control-001/solr.xml
   [junit4]   2> 721919 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[93F883ACB19AA413]) [    ] 
o.a.s.c.SolrXmlConfig MBean server found: 
com.sun.jmx.mbeanserver.JmxMBeanServer@456c89fd, but no JMX reporters were 
configured - adding default JMX reporter.
   [junit4]   2> 722929 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[93F883ACB19AA413]) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51856/solr
   [junit4]   2> 724254 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[93F883ACB19AA413]) 
[n:127.0.0.1:35965_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 724254 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[93F883ACB19AA413]) 
[n:127.0.0.1:35965_    ] o.a.s.c.OverseerElectionContext I am going to be the 
leader 127.0.0.1:35965_
   [junit4]   2> 724257 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[93F883ACB19AA413]) 
[n:127.0.0.1:35965_    ] o.a.s.c.Overseer Overseer 
(id=98770511820095492-127.0.0.1:35965_-n_0000000000) starting
   [junit4]   2> 724301 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[93F883ACB19AA413]) 
[n:127.0.0.1:35965_    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:35965_
   [junit4]   2> 725440 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[93F883ACB19AA413]) 
[n:127.0.0.1:35965_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@456c89fd
   [junit4]   2> 725592 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[93F883ACB19AA413]) 
[n:127.0.0.1:35965_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@456c89fd
   [junit4]   2> 725592 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[93F883ACB19AA413]) 
[n:127.0.0.1:35965_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@456c89fd
   [junit4]   2> 725621 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[93F883ACB19AA413]) 
[n:127.0.0.1:35965_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions 
underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_93F883ACB19AA413-001/control-001/cores
   [junit4]   2> 725653 INFO  
(zkCallback-238-thread-1-processing-n:127.0.0.1:35965_) [n:127.0.0.1:35965_    
] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 725861 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[93F883ACB19AA413]) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 725862 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[93F883ACB19AA413]) [    ] 
o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:51856/solr ready
   [junit4]   2> 725923 INFO  (qtp576116407-1734) [n:127.0.0.1:35965_    ] 
o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params 
replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:35965_&wt=javabin&version=2
 and sendToOCPQueue=true
   [junit4]   2> 725969 INFO  
(OverseerThreadFactory-621-thread-1-processing-n:127.0.0.1:35965_) 
[n:127.0.0.1:35965_    ] o.a.s.c.CreateCollectionCmd Create collection 
control_collection
   [junit4]   2> 726265 INFO  (qtp576116407-1732) [n:127.0.0.1:35965_    ] 
o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 726266 INFO  (qtp576116407-1732) [n:127.0.0.1:35965_    ] 
o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 
transient cores
   [junit4]   2> 726421 INFO  
(zkCallback-238-thread-1-processing-n:127.0.0.1:35965_) [n:127.0.0.1:35965_    
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/control_collection/state.json] for collection 
[control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 727430 INFO  (qtp576116407-1732) [n:127.0.0.1:35965_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 727779 INFO  (qtp576116407-1732) [n:127.0.0.1:35965_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 728934 INFO  (qtp576116407-1732) [n:127.0.0.1:35965_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 729646 INFO  (qtp576116407-1732) [n:127.0.0.1:35965_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' 
using configuration from collection control_collection, trusted=true
   [junit4]   2> 729651 INFO  (qtp576116407-1732) [n:127.0.0.1:35965_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.core.control_collection.shard1.replica_n1' (registry 
'solr.core.control_collection.shard1.replica_n1') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@456c89fd
   [junit4]   2> 729778 INFO  (qtp576116407-1732) [n:127.0.0.1:35965_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.HdfsDirectoryFactory 
solr.hdfs.home=hdfs://localhost:37663/solr_hdfs_home
   [junit4]   2> 729778 INFO  (qtp576116407-1732) [n:127.0.0.1:35965_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 729778 INFO  (qtp576116407-1732) [n:127.0.0.1:35965_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 729778 INFO  (qtp576116407-1732) [n:127.0.0.1:35965_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore 
at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_93F883ACB19AA413-001/control-001/cores/control_collection_shard1_replica_n1],
 
dataDir=[hdfs://localhost:37663/solr_hdfs_home/control_collection/core_node2/data/]
   [junit4]   2> 729780 INFO  (qtp576116407-1732) [n:127.0.0.1:35965_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:37663/solr_hdfs_home/control_collection/core_node2/data/snapshot_metadata
   [junit4]   2> 730011 INFO  (qtp576116407-1732) [n:127.0.0.1:35965_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct 
memory allocation set to [true]
   [junit4]   2> 730011 INFO  (qtp576116407-1732) [n:127.0.0.1:35965_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of 
[4194304] will allocate [1] slabs and use ~[4194304] bytes
   [junit4]   2> 730011 INFO  (qtp576116407-1732) [n:127.0.0.1:35965_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.HdfsDirectoryFactory Creating new global HDFS BlockCache
   [junit4]   2> 739237 INFO  (qtp576116407-1732) [n:127.0.0.1:35965_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 739521 INFO  (qtp576116407-1732) [n:127.0.0.1:35965_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:37663/solr_hdfs_home/control_collection/core_node2/data
   [junit4]   2> 740593 INFO  (qtp576116407-1732) [n:127.0.0.1:35965_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:37663/solr_hdfs_home/control_collection/core_node2/data/index
   [junit4]   2> 740995 INFO  (qtp576116407-1732) [n:127.0.0.1:35965_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct 
memory allocation set to [true]
   [junit4]   2> 740996 INFO  (qtp576116407-1732) [n:127.0.0.1:35965_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of 
[4194304] will allocate [1] slabs and use ~[4194304] bytes
   [junit4]   2> 741049 INFO  (qtp576116407-1732) [n:127.0.0.1:35965_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 741049 INFO  (qtp576116407-1732) [n:127.0.0.1:35965_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.MockRandomMergePolicy: 
org.apache.lucene.index.MockRandomMergePolicy@3ba56b3a
   [junit4]   2> 744561 INFO  (Block report processor) [    ] BlockStateChange 
BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59042 is added to 
blk_1073741825_1001{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-bab6ae00-2ba5-4efb-9333-2d403c9fc9d9:NORMAL:127.0.0.1:59042|RBW],
 
ReplicaUC[[DISK]DS-6a655397-f967-4ddb-8113-40d611b6dddb:NORMAL:127.0.0.1:58442|RBW]]}
 size 72
   [junit4]   2> 744562 INFO  (Block report processor) [    ] BlockStateChange 
BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:58442 is added to 
blk_1073741825_1001 size 72
   [junit4]   2> 745063 WARN  (qtp576116407-1732) [n:127.0.0.1:35965_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = 
requestHandler,name = /dump,class = DumpRequestHandler,attributes = 
{initParams=a, name=/dump, class=DumpRequestHandler},args = 
{defaults={a=A,b=B}}}
   [junit4]   2> 745521 INFO  (qtp576116407-1732) [n:127.0.0.1:35965_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.u.UpdateHandler Using UpdateLog implementation: 
org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 745521 INFO  (qtp576116407-1732) [n:127.0.0.1:35965_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 745521 INFO  (qtp576116407-1732) [n:127.0.0.1:35965_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 747033 INFO  (qtp576116407-1732) [n:127.0.0.1:35965_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 747033 INFO  (qtp576116407-1732) [n:127.0.0.1:35965_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 747072 INFO  (qtp576116407-1732) [n:127.0.0.1:35965_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.MockRandomMergePolicy: 
org.apache.lucene.index.MockRandomMergePolicy@fb53b6e
   [junit4]   2> 749846 INFO  (qtp576116407-1732) [n:127.0.0.1:35965_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@24fa1a67[control_collection_shard1_replica_n1] main]
   [junit4]   2> 749855 INFO  (qtp576116407-1732) [n:127.0.0.1:35965_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf1
   [junit4]   2> 749855 INFO  (qtp576116407-1732) [n:127.0.0.1:35965_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 749858 INFO  (qtp576116407-1732) [n:127.0.0.1:35965_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 749860 INFO  (qtp576116407-1732) [n:127.0.0.1:35965_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.u.UpdateLog Could not find max version in index or recent updates, using 
new clock 1580328220730851328
   [junit4]   2> 749936 INFO  (qtp576116407-1732) [n:127.0.0.1:35965_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 749936 INFO  (qtp576116407-1732) [n:127.0.0.1:35965_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 749936 INFO  (qtp576116407-1732) [n:127.0.0.1:35965_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.SyncStrategy Sync replicas to 
http://127.0.0.1:35965/control_collection_shard1_replica_n1/
   [junit4]   2> 749936 INFO  (qtp576116407-1732) [n:127.0.0.1:35965_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 749936 INFO  (qtp576116407-1732) [n:127.0.0.1:35965_ 
c:control_collection s:shard1  x:control_collection_shard1_re

[...truncated too long message...]

DistributedZkTest]
   [junit4]    > Caused by: 
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at http://127.0.0.1:33552: Cannot unload non-existent core 
[multiunload12]
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([93F883ACB19AA413]:0)
   [junit4]    >        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:627)
   [junit4]    >        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
   [junit4]    >        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
   [junit4]    >        at 
org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
   [junit4]    >        at 
org.apache.solr.cloud.UnloadDistributedZkTest.lambda$testUnloadLotsOfCores$0(UnloadDistributedZkTest.java:362)
   [junit4]    >        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]    >        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]    >        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]    >        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 6578904 WARN  
(SUITE-HdfsUnloadDistributedZkTest-seed#[93F883ACB19AA413]-worker) [    ] 
o.a.h.h.s.d.DirectoryScanner DirectoryScanner: shutdown has been called
   [junit4]   2> 6579641 INFO  
(SUITE-HdfsUnloadDistributedZkTest-seed#[93F883ACB19AA413]-worker) [    ] 
o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
   [junit4]   2> 6579773 WARN  (DataNode: 
[[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_93F883ACB19AA413-001/tempDir-001/hdfsBaseDir/data/data3/,
 
[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_93F883ACB19AA413-001/tempDir-001/hdfsBaseDir/data/data4/]]
  heartbeating to localhost/127.0.0.1:58126) [    ] 
o.a.h.h.s.d.IncrementalBlockReportManager IncrementalBlockReportManager 
interrupted
   [junit4]   2> 6579773 WARN  (DataNode: 
[[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_93F883ACB19AA413-001/tempDir-001/hdfsBaseDir/data/data3/,
 
[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_93F883ACB19AA413-001/tempDir-001/hdfsBaseDir/data/data4/]]
  heartbeating to localhost/127.0.0.1:58126) [    ] o.a.h.h.s.d.DataNode Ending 
block pool service for: Block pool BP-2105895430-10.41.0.5-1507123971539 
(Datanode Uuid 211baa6c-b20b-453c-9b62-0d917ca38c71) service to 
localhost/127.0.0.1:58126
   [junit4]   2> 6579774 WARN  
(SUITE-HdfsUnloadDistributedZkTest-seed#[93F883ACB19AA413]-worker) [    ] 
o.a.h.h.s.d.DirectoryScanner DirectoryScanner: shutdown has been called
   [junit4]   2> 6580041 INFO  
(SUITE-HdfsUnloadDistributedZkTest-seed#[93F883ACB19AA413]-worker) [    ] 
o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
   [junit4]   2> 6580086 WARN  (2064534012@qtp-608204702-1 - Acceptor0 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46878) [    ] 
o.a.h.h.HttpServer2 HttpServer Acceptor: isRunning is false. Rechecking.
   [junit4]   2> 6580086 WARN  (2064534012@qtp-608204702-1 - Acceptor0 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46878) [    ] 
o.a.h.h.HttpServer2 HttpServer Acceptor: isRunning is false
   [junit4]   2> 6580127 WARN  (DataNode: 
[[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_93F883ACB19AA413-001/tempDir-001/hdfsBaseDir/data/data1/,
 
[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_93F883ACB19AA413-001/tempDir-001/hdfsBaseDir/data/data2/]]
  heartbeating to localhost/127.0.0.1:58126) [    ] 
o.a.h.h.s.d.IncrementalBlockReportManager IncrementalBlockReportManager 
interrupted
   [junit4]   2> 6580127 WARN  (DataNode: 
[[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_93F883ACB19AA413-001/tempDir-001/hdfsBaseDir/data/data1/,
 
[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_93F883ACB19AA413-001/tempDir-001/hdfsBaseDir/data/data2/]]
  heartbeating to localhost/127.0.0.1:58126) [    ] o.a.h.h.s.d.DataNode Ending 
block pool service for: Block pool BP-2105895430-10.41.0.5-1507123971539 
(Datanode Uuid 116c9784-0fba-48ba-9553-3635053949d3) service to 
localhost/127.0.0.1:58126
   [junit4]   2> 6580240 INFO  
(SUITE-HdfsUnloadDistributedZkTest-seed#[93F883ACB19AA413]-worker) [    ] 
o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
   [junit4]   2> 6581736 INFO  
(SUITE-HdfsUnloadDistributedZkTest-seed#[93F883ACB19AA413]-worker) [    ] 
o.a.s.SolrTestCaseJ4 ------------------------------------------------------- 
Done waiting for tracked resources to be released
   [junit4]   2> Oct 04, 2017 1:37:45 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 2 leaked 
thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): 
{multiDefault=PostingsFormat(name=Memory), a_t=FST50, 
id=PostingsFormat(name=Memory), text=BlockTreeOrds(blocksize=128)}, 
docValues:{other_tl1=DocValuesFormat(name=Asserting), 
range_facet_l_dv=DocValuesFormat(name=Memory), 
_version_=DocValuesFormat(name=Asserting), 
intDefault=DocValuesFormat(name=Asserting), 
id_i1=DocValuesFormat(name=Lucene70), a_i1=DocValuesFormat(name=Asserting), 
range_facet_i_dv=DocValuesFormat(name=Asserting), 
intDvoDefault=DocValuesFormat(name=Direct), 
range_facet_l=DocValuesFormat(name=Asserting), 
timestamp=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=222, 
maxMBSortInHeap=7.413352655513397, sim=RandomSimilarity(queryNorm=true): {}, 
locale=id, timezone=Canada/Saskatchewan
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 
1.8.0_144 (64-bit)/cpus=4,threads=2,free=343507552,total=528482304
   [junit4]   2> NOTE: All tests run in this JVM: [TestCharFilters, 
TestStressUserVersions, DocValuesMissingTest, ClusterStateUpdateTest, 
BlockJoinFacetSimpleTest, TestGraphTermsQParserPlugin, TestManagedSchemaAPI, 
PropertiesRequestHandlerTest, AliasIntegrationTest, 
TestManagedSynonymGraphFilterFactory, RemoteQueryErrorTest, 
TestRandomCollapseQParserPlugin, TestPartialUpdateDeduplication, 
TestSegmentSorting, TestRawResponseWriter, TestCursorMarkWithoutUniqueKey, 
HdfsDirectoryTest, TestFileDictionaryLookup, TestPullReplica, 
ShardRoutingCustomTest, SchemaVersionSpecificBehaviorTest, 
SolrSlf4jReporterTest, CdcrUpdateLogTest, CollectionReloadTest, 
SolrMetricReporterTest, ConvertedLegacyTest, HdfsSyncSliceTest, 
RestartWhileUpdatingTest, DisMaxRequestHandlerTest, 
SpellCheckCollatorWithCollapseTest, HdfsRecoverLeaseTest, 
DocValuesNotIndexedTest, ZkControllerTest, TestHashPartitioner, 
TestFuzzyAnalyzedSuggestions, TestPerFieldSimilarityWithDefaultOverride, 
HttpSolrCallGetCoreTest, ScriptEngineTest, TestSolrQueryResponse, 
DocExpirationUpdateProcessorFactoryTest, TestDistribDocBasedVersion, 
TestLRUStatsCache, DistributedExpandComponentTest, IndexSchemaTest, 
MoveReplicaTest, TestNumericTokenStream, TestBM25SimilarityFactory, 
TestTrackingShardHandlerFactory, TestRequestForwarding, 
TestSimpleQParserPlugin, TestHdfsUpdateLog, CoreAdminHandlerTest, 
TermVectorComponentTest, TestConfigSetProperties, 
PreAnalyzedUpdateProcessorTest, TestReplicationHandlerBackup, 
TestGraphMLResponseWriter, TestExpandComponent, BasicFunctionalityTest, 
TestCloudPivotFacet, HdfsUnloadDistributedZkTest]
   [junit4] Completed [600/733 (6!)] on J1 in 298.30s, 1 test, 1 error <<< 
FAILURES!

[...truncated 47035 lines...]
-documentation-lint:
    [jtidy] Checking for broken html (such as invalid tags)...
   [delete] Deleting directory 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/build/jtidy_tmp
     [echo] Checking for broken links...
     [exec] 
     [exec] Crawl/parse...
     [exec] 
     [exec] Verify...
     [exec] 
     [exec] 
file:///x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/docs/quickstart.html
     [exec]   BAD EXTERNAL LINK: 
https://lucene.apache.org/solr/guide/solr-tutorial.html
     [exec] 
     [exec] Broken javadocs links were found! Common root causes:
     [exec] * A typo of some sort for manually created links.
     [exec] * Public methods referencing non-public classes in their signature.

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/build.xml:834:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/build.xml:101:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build.xml:669:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build.xml:682:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/common-build.xml:2570:
 exec returned: 1

Total time: 908 minutes 39 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to