[JENKINS] Lucene-Solr-NightlyTests-5.2 - Build # 5 - Still Failing

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.2/5/

4 tests failed.
REGRESSION:  org.apache.solr.cloud.MultiThreadedOCPTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=6237, 
name=parallelCoreAdminExecutor-2470-thread-14, state=RUNNABLE, 
group=TGRP-MultiThreadedOCPTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=6237, 
name=parallelCoreAdminExecutor-2470-thread-14, state=RUNNABLE, 
group=TGRP-MultiThreadedOCPTest]
        at 
__randomizedtesting.SeedInfo.seed([1E0D9EBC8B4DDDE1:9659A16625B1B019]:0)
Caused by: java.lang.AssertionError: Too many closes on SolrCore
        at __randomizedtesting.SeedInfo.seed([1E0D9EBC8B4DDDE1]:0)
        at org.apache.solr.core.SolrCore.close(SolrCore.java:1157)
        at org.apache.solr.common.util.IOUtils.closeQuietly(IOUtils.java:31)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:652)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:611)
        at 
org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:628)
        at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:213)
        at 
org.apache.solr.handler.admin.CoreAdminHandler$ParallelCoreAdminHandlerThread.run(CoreAdminHandler.java:1249)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  
junit.framework.TestSuite.org.apache.solr.search.TestMaxScoreQueryParser

Error Message:


Stack Trace:
java.lang.AssertionError
        at 
org.apache.lucene.util.TestRuleTemporaryFilesCleanup.before(TestRuleTemporaryFilesCleanup.java:113)
        at 
com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.before(TestRuleAdapter.java:26)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:35)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  
junit.framework.TestSuite.org.apache.solr.search.TestMaxScoreQueryParser

Error Message:
45 threads leaked from SUITE scope at 
org.apache.solr.search.TestMaxScoreQueryParser:     1) Thread[id=15807, 
name=coreZkRegister-4049-thread-71-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:888)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
         at java.util.concurrent.FutureTask.run(FutureTask.java:262)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)    2) Thread[id=15832, 
name=coreZkRegister-4049-thread-96-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:888)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
         at java.util.concurrent.FutureTask.run(FutureTask.java:262)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)    3) Thread[id=15823, 
name=coreZkRegister-4049-thread-87-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:888)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
         at java.util.concurrent.FutureTask.run(FutureTask.java:262)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)    4) Thread[id=15827, 
name=coreZkRegister-4049-thread-91-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:888)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
         at java.util.concurrent.FutureTask.run(FutureTask.java:262)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)    5) Thread[id=15806, 
name=coreZkRegister-4049-thread-70-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:888)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
         at java.util.concurrent.FutureTask.run(FutureTask.java:262)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)    6) Thread[id=15796, 
name=zkCallback-1213-thread-28-processing-{node_name=127.0.0.1:53961_}-EventThread,
 state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)    
7) Thread[id=15816, 
name=coreZkRegister-4049-thread-80-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:888)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
         at java.util.concurrent.FutureTask.run(FutureTask.java:262)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)    8) Thread[id=15841, 
name=zkCallback-1207-thread-99, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)    9) Thread[id=15794, 
name=zkCallback-1217-thread-29-processing-{node_name=127.0.0.1:38044_}-EventThread,
 state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)   
10) Thread[id=15802, 
name=OverseerCollectionProcessor-93898657861468232-127.0.0.1:57064_-n_0000000021,
 state=TIMED_WAITING, group=Overseer collection creation process.]         at 
java.lang.Object.wait(Native Method)         at 
org.apache.solr.cloud.DistributedQueue$LatchWatcher.await(DistributedQueue.java:276)
         at 
org.apache.solr.cloud.DistributedQueue.getChildren(DistributedQueue.java:320)   
      at 
org.apache.solr.cloud.DistributedQueue.peekTopN(DistributedQueue.java:447)      
   at 
org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:301)
         at java.lang.Thread.run(Thread.java:745)   11) Thread[id=15820, 
name=coreZkRegister-4049-thread-84-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:888)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
         at java.util.concurrent.FutureTask.run(FutureTask.java:262)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)   12) Thread[id=15791, 
name=zkCallback-1210-thread-22-processing-{node_name=127.0.0.1:39037_}-SendThread(127.0.0.1:55012),
 state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at 
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at 
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)         at 
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)         at 
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)         at 
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
         at 
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)   13) 
Thread[id=15834, name=coreZkRegister-4049-thread-98, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)   14) Thread[id=15836, 
name=coreZkRegister-4049-thread-100, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)   15) Thread[id=15808, 
name=coreZkRegister-4049-thread-72-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:888)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
         at java.util.concurrent.FutureTask.run(FutureTask.java:262)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)   16) Thread[id=15814, 
name=coreZkRegister-4049-thread-78-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:888)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
         at java.util.concurrent.FutureTask.run(FutureTask.java:262)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)   17) Thread[id=15801, 
name=OverseerStateUpdate-93898657861468232-127.0.0.1:57064_-n_0000000021, 
state=WAITING, group=Overseer state updater.]         at 
org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:528)
         at 
org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:269)
         at 
org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:152)       
  at java.lang.Thread.run(Thread.java:745)   18) Thread[id=15830, 
name=coreZkRegister-4049-thread-94, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)   19) Thread[id=15803, 
name=OverseerHdfsCoreFailoverThread-93898657861468232-127.0.0.1:57064_-n_0000000021,
 state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:139)
         at java.lang.Thread.run(Thread.java:745)   20) Thread[id=15828, 
name=coreZkRegister-4049-thread-92-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:888)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
         at java.util.concurrent.FutureTask.run(FutureTask.java:262)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)   21) Thread[id=15812, 
name=coreZkRegister-4049-thread-76-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:888)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
         at java.util.concurrent.FutureTask.run(FutureTask.java:262)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)   22) Thread[id=15792, 
name=zkCallback-1210-thread-22-processing-{node_name=127.0.0.1:39037_}-EventThread,
 state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)   
23) Thread[id=15810, 
name=coreZkRegister-4049-thread-74-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:888)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
         at java.util.concurrent.FutureTask.run(FutureTask.java:262)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)   24) Thread[id=15821, 
name=coreZkRegister-4049-thread-85-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:888)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
         at java.util.concurrent.FutureTask.run(FutureTask.java:262)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)   25) Thread[id=15819, 
name=coreZkRegister-4049-thread-83-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:888)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
         at java.util.concurrent.FutureTask.run(FutureTask.java:262)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)   26) Thread[id=15833, 
name=coreZkRegister-4049-thread-97, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)   27) Thread[id=15824, 
name=coreZkRegister-4049-thread-88-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:888)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
         at java.util.concurrent.FutureTask.run(FutureTask.java:262)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)   28) Thread[id=15795, 
name=zkCallback-1213-thread-28-processing-{node_name=127.0.0.1:53961_}-SendThread(127.0.0.1:55012),
 state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at 
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at 
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)         at 
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)         at 
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)         at 
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
         at 
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)   29) 
Thread[id=15811, 
name=coreZkRegister-4049-thread-75-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:888)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
         at java.util.concurrent.FutureTask.run(FutureTask.java:262)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)   30) Thread[id=15835, 
name=coreZkRegister-4049-thread-99, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)   31) Thread[id=15804, 
name=coreZkRegister-4049-thread-68-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:888)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
         at java.util.concurrent.FutureTask.run(FutureTask.java:262)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)   32) Thread[id=15829, 
name=coreZkRegister-4049-thread-93-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:888)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
         at java.util.concurrent.FutureTask.run(FutureTask.java:262)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)   33) Thread[id=15840, 
name=zkCallback-1207-thread-98, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)   34) Thread[id=15817, 
name=coreZkRegister-4049-thread-81-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:888)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
         at java.util.concurrent.FutureTask.run(FutureTask.java:262)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)   35) Thread[id=15822, 
name=coreZkRegister-4049-thread-86-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:888)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
         at java.util.concurrent.FutureTask.run(FutureTask.java:262)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)   36) Thread[id=15805, 
name=coreZkRegister-4049-thread-69-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:888)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
         at java.util.concurrent.FutureTask.run(FutureTask.java:262)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)   37) Thread[id=15809, 
name=coreZkRegister-4049-thread-73-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:682)
         at 
org.apache.solr.common.cloud.ZkStateReader.getLeaderUrl(ZkStateReader.java:651) 
        at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:964)  
       at org.apache.solr.cloud.ZkController.register(ZkController.java:901)    
     at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
         at java.util.concurrent.FutureTask.run(FutureTask.java:262)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)   38) Thread[id=15793, 
name=zkCallback-1217-thread-29-processing-{node_name=127.0.0.1:38044_}-SendThread(127.0.0.1:55012),
 state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at 
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at 
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)         at 
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)         at 
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)         at 
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
         at 
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)   39) 
Thread[id=15825, 
name=coreZkRegister-4049-thread-89-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:888)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
         at java.util.concurrent.FutureTask.run(FutureTask.java:262)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)   40) Thread[id=15831, 
name=coreZkRegister-4049-thread-95-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:888)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
         at java.util.concurrent.FutureTask.run(FutureTask.java:262)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)   41) Thread[id=15818, 
name=coreZkRegister-4049-thread-82-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:888)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
         at java.util.concurrent.FutureTask.run(FutureTask.java:262)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)   42) Thread[id=15815, 
name=coreZkRegister-4049-thread-79-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:888)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
         at java.util.concurrent.FutureTask.run(FutureTask.java:262)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)   43) Thread[id=15813, 
name=coreZkRegister-4049-thread-77-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:888)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
         at java.util.concurrent.FutureTask.run(FutureTask.java:262)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)   44) Thread[id=15839, 
name=zkCallback-1207-thread-97, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)   45) Thread[id=15826, 
name=coreZkRegister-4049-thread-90-processing-{node_name=127.0.0.1:57064_}, 
state=BLOCKED, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:888)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
         at java.util.concurrent.FutureTask.run(FutureTask.java:262)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 45 threads leaked from 
SUITE scope at org.apache.solr.search.TestMaxScoreQueryParser: 
   1) Thread[id=15807, 
name=coreZkRegister-4049-thread-71-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
        at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
        at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
        at 
org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
        at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
        at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=15832, 
name=coreZkRegister-4049-thread-96-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
        at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
        at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
        at 
org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
        at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
        at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=15823, 
name=coreZkRegister-4049-thread-87-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
        at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
        at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
        at 
org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
        at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
        at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=15827, 
name=coreZkRegister-4049-thread-91-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
        at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
        at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
        at 
org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
        at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
        at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=15806, 
name=coreZkRegister-4049-thread-70-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
        at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
        at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
        at 
org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
        at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
        at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=15796, 
name=zkCallback-1213-thread-28-processing-{node_name=127.0.0.1:53961_}-EventThread,
 state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
   7) Thread[id=15816, 
name=coreZkRegister-4049-thread-80-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
        at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
        at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
        at 
org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
        at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
        at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=15841, name=zkCallback-1207-thread-99, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
   9) Thread[id=15794, 
name=zkCallback-1217-thread-29-processing-{node_name=127.0.0.1:38044_}-EventThread,
 state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
  10) Thread[id=15802, 
name=OverseerCollectionProcessor-93898657861468232-127.0.0.1:57064_-n_0000000021,
 state=TIMED_WAITING, group=Overseer collection creation process.]
        at java.lang.Object.wait(Native Method)
        at 
org.apache.solr.cloud.DistributedQueue$LatchWatcher.await(DistributedQueue.java:276)
        at 
org.apache.solr.cloud.DistributedQueue.getChildren(DistributedQueue.java:320)
        at 
org.apache.solr.cloud.DistributedQueue.peekTopN(DistributedQueue.java:447)
        at 
org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:301)
        at java.lang.Thread.run(Thread.java:745)
  11) Thread[id=15820, 
name=coreZkRegister-4049-thread-84-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
        at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
        at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
        at 
org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
        at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
        at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
  12) Thread[id=15791, 
name=zkCallback-1210-thread-22-processing-{node_name=127.0.0.1:39037_}-SendThread(127.0.0.1:55012),
 state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
        at 
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
  13) Thread[id=15834, name=coreZkRegister-4049-thread-98, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
  14) Thread[id=15836, name=coreZkRegister-4049-thread-100, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
  15) Thread[id=15808, 
name=coreZkRegister-4049-thread-72-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
        at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
        at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
        at 
org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
        at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
        at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
  16) Thread[id=15814, 
name=coreZkRegister-4049-thread-78-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
        at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
        at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
        at 
org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
        at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
        at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
  17) Thread[id=15801, 
name=OverseerStateUpdate-93898657861468232-127.0.0.1:57064_-n_0000000021, 
state=WAITING, group=Overseer state updater.]
        at 
org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:528)
        at 
org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:269)
        at 
org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:152)
        at java.lang.Thread.run(Thread.java:745)
  18) Thread[id=15830, name=coreZkRegister-4049-thread-94, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
  19) Thread[id=15803, 
name=OverseerHdfsCoreFailoverThread-93898657861468232-127.0.0.1:57064_-n_0000000021,
 state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:139)
        at java.lang.Thread.run(Thread.java:745)
  20) Thread[id=15828, 
name=coreZkRegister-4049-thread-92-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
        at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
        at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
        at 
org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
        at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
        at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
  21) Thread[id=15812, 
name=coreZkRegister-4049-thread-76-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
        at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
        at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
        at 
org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
        at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
        at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
  22) Thread[id=15792, 
name=zkCallback-1210-thread-22-processing-{node_name=127.0.0.1:39037_}-EventThread,
 state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
  23) Thread[id=15810, 
name=coreZkRegister-4049-thread-74-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
        at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
        at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
        at 
org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
        at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
        at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
  24) Thread[id=15821, 
name=coreZkRegister-4049-thread-85-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
        at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
        at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
        at 
org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
        at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
        at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
  25) Thread[id=15819, 
name=coreZkRegister-4049-thread-83-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
        at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
        at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
        at 
org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
        at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
        at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
  26) Thread[id=15833, name=coreZkRegister-4049-thread-97, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
  27) Thread[id=15824, 
name=coreZkRegister-4049-thread-88-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
        at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
        at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
        at 
org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
        at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
        at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
  28) Thread[id=15795, 
name=zkCallback-1213-thread-28-processing-{node_name=127.0.0.1:53961_}-SendThread(127.0.0.1:55012),
 state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
        at 
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
  29) Thread[id=15811, 
name=coreZkRegister-4049-thread-75-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
        at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
        at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
        at 
org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
        at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
        at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
  30) Thread[id=15835, name=coreZkRegister-4049-thread-99, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
  31) Thread[id=15804, 
name=coreZkRegister-4049-thread-68-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
        at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
        at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
        at 
org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
        at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
        at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
  32) Thread[id=15829, 
name=coreZkRegister-4049-thread-93-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
        at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
        at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
        at 
org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
        at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
        at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
  33) Thread[id=15840, name=zkCallback-1207-thread-98, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
  34) Thread[id=15817, 
name=coreZkRegister-4049-thread-81-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
        at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
        at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
        at 
org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
        at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
        at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
  35) Thread[id=15822, 
name=coreZkRegister-4049-thread-86-processing-{node_name=127.0.0.1:57064_}, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
        at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
        at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
        at 
org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
        at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
        at org.apache.solr.cloud.ZkControl

[...truncated too long message...]

java:603)
   [junit4]   2>        at java.io.PrintWriter.println(PrintWriter.java:756)
   [junit4]   2>        at 
java.lang.Throwable$WrappedPrintWriter.println(Throwable.java:764)
   [junit4]   2>        at 
java.lang.Throwable.printStackTrace(Throwable.java:658)
   [junit4]   2>        at 
java.lang.Throwable.printStackTrace(Throwable.java:721)
   [junit4]   2>        at 
org.apache.solr.common.SolrException.toStr(SolrException.java:170)
   [junit4]   2>        at 
org.apache.solr.common.SolrException.log(SolrException.java:133)
   [junit4]   2>        at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:164)
   [junit4]   2>        at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:646)
   [junit4]   2>        at 
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:417)
   [junit4]   2>        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
   [junit4]   2>        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
   [junit4]   2>        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
   [junit4]   2>        at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:105)
   [junit4]   2>        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
   [junit4]   2>        at 
org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
   [junit4]   2>        at 
org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364)
   [junit4]   2>        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
   [junit4]   2>        at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
   [junit4]   2>        at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
   [junit4]   2>        at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
   [junit4]   2>        at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
   [junit4]   2>        at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2>        at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
   [junit4]   2>        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2>        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
   [junit4]   2>        at 
org.eclipse.jetty.server.Server.handle(Server.java:497)
   [junit4]   2>        at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
   [junit4]   2> 
   [junit4]   2> 10154996 T15161 n:127.0.0.1:57064_ 
oejs.ServletHandler.doHandle WARN Error for /admin/cores 
java.lang.OutOfMemoryError: Java heap space
   [junit4]   2>        at java.util.Arrays.copyOf(Arrays.java:2367)
   [junit4]   2>        at 
java.io.CharArrayWriter.write(CharArrayWriter.java:122)
   [junit4]   2>        at java.io.PrintWriter.write(PrintWriter.java:456)
   [junit4]   2>        at java.io.PrintWriter.write(PrintWriter.java:473)
   [junit4]   2>        at java.io.PrintWriter.print(PrintWriter.java:603)
   [junit4]   2>        at java.io.PrintWriter.println(PrintWriter.java:756)
   [junit4]   2>        at 
java.lang.Throwable$WrappedPrintWriter.println(Throwable.java:764)
   [junit4]   2>        at 
java.lang.Throwable.printStackTrace(Throwable.java:658)
   [junit4]   2>        at 
java.lang.Throwable.printStackTrace(Throwable.java:721)
   [junit4]   2>        at 
org.apache.solr.common.SolrException.toStr(SolrException.java:170)
   [junit4]   2>        at 
org.apache.solr.common.SolrException.log(SolrException.java:133)
   [junit4]   2>        at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:164)
   [junit4]   2>        at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:646)
   [junit4]   2>        at 
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:417)
   [junit4]   2>        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
   [junit4]   2>        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
   [junit4]   2>        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
   [junit4]   2>        at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:105)
   [junit4]   2>        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
   [junit4]   2>        at 
org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
   [junit4]   2>        at 
org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364)
   [junit4]   2>        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
   [junit4]   2>        at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
   [junit4]   2>        at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
   [junit4]   2>        at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
   [junit4]   2>        at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
   [junit4]   2>        at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2>        at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
   [junit4]   2>        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2>        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
   [junit4]   2>        at 
org.eclipse.jetty.server.Server.handle(Server.java:497)
   [junit4]   2>        at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
   [junit4]   2> 
   [junit4] May 28, 2015 12:31:23 AM 
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus$LoggingSubscriberExceptionHandler
 handleException
   [junit4] SEVERE: Could not dispatch event: 
com.carrotsearch.ant.tasks.junit4.listeners.TextReport@105cf2a8 to public void 
com.carrotsearch.ant.tasks.junit4.listeners.TextReport.onSuiteResult(com.carrotsearch.ant.tasks.junit4.events.aggregated.AggregatedSuiteResultEvent)
   [junit4] May 28, 2015 12:31:23 AM 
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus$LoggingSubscriberExceptionHandler
 handleException
   [junit4] SEVERE: Could not dispatch event: 
com.carrotsearch.ant.tasks.junit4.listeners.TextReport@22ae4c81 to public void 
com.carrotsearch.ant.tasks.junit4.listeners.TextReport.onSuiteResult(com.carrotsearch.ant.tasks.junit4.events.aggregated.AggregatedSuiteResultEvent)
   [junit4] May 28, 2015 12:31:23 AM 
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus$LoggingSubscriberExceptionHandler
 handleException
   [junit4] SEVERE: Could not dispatch event: 
com.carrotsearch.ant.tasks.junit4.listeners.TextReport@5139b168 to public void 
com.carrotsearch.ant.tasks.junit4.listeners.TextReport.onSuiteResult(com.carrotsearch.ant.tasks.junit4.events.aggregated.AggregatedSuiteResultEvent)
   [junit4] Suite: 
org.apache.solr.cloud.VMParamsZkACLAndCredentialsProvidersTest
   [junit4] Completed [496/497] on J1 in 0.93s, 4 tests, 4 skipped
   [junit4] 
   [junit4] Suite: org.apache.solr.search.TestCustomSort
   [junit4] Completed [497/497] on J1 in 0.01s, 1 test, 1 skipped
   [junit4] 
   [junit4] JVM J1: stdout was not empty, see: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/temp/junit4-J1-20150527_214030_137.sysout
   [junit4] >>> JVM J1: stdout (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/heapdumps/java_pid1911.hprof
 ...
   [junit4] Heap dump file created [646867182 bytes in 6.369 secs]
   [junit4] <<< JVM J1: EOF ----
   [junit4] 
   [junit4] Tests with failures:
   [junit4]   - org.apache.solr.cloud.MultiThreadedOCPTest.test
   [junit4]   - org.apache.solr.cloud.CollectionsAPIDistributedZkTest (suite)
   [junit4] 
   [junit4] 
   [junit4] JVM J0:     0.73 ..  3406.48 =  3405.75s
   [junit4] JVM J1:     0.95 .. 10255.73 = 10254.79s
   [junit4] JVM J2:     0.95 ..  3397.47 =  3396.52s
   [junit4] Execution time total: 2 hours 50 minutes 56 seconds
   [junit4] Tests summary: 497 suites, 1967 tests, 4 suite-level errors, 1 
error, 56 ignored (10 assumptions)

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/build.xml:543: 
The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/build.xml:484: 
The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/build.xml:61: 
The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/extra-targets.xml:39:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build.xml:229:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/common-build.xml:511:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/common-build.xml:1433:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/common-build.xml:991:
 There were test failures: 497 suites, 1967 tests, 4 suite-level errors, 1 
error, 56 ignored (10 assumptions)

Total time: 284 minutes 19 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-5.2 #3
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 1583434417 bytes
Compression is 0.0%
Took 4 min 46 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org