[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 801 - Still Failing

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/801/

7 tests failed.
REGRESSION:  org.apache.solr.cloud.CdcrReplicationHandlerTest.doTest

Error Message:
Captured an uncaught exception in thread: Thread[id=18777, 
name=RecoveryThread-source_collection_shard1_replica2, state=RUNNABLE, 
group=TGRP-CdcrReplicationHandlerTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=18777, 
name=RecoveryThread-source_collection_shard1_replica2, state=RUNNABLE, 
group=TGRP-CdcrReplicationHandlerTest]
Caused by: org.apache.solr.common.cloud.ZooKeeperException: 
        at __randomizedtesting.SeedInfo.seed([AD79661DF28559C5]:0)
        at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:234)
Caused by: org.apache.solr.common.SolrException: java.io.FileNotFoundException: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationHandlerTest_AD79661DF28559C5-001/jetty-002/cores/source_collection_shard1_replica2/data/tlog/tlog.0000000000000000008.1513034951642054656
 (No such file or directory)
        at 
org.apache.solr.update.CdcrTransactionLog.reopenOutputStream(CdcrTransactionLog.java:244)
        at 
org.apache.solr.update.CdcrTransactionLog.incref(CdcrTransactionLog.java:173)
        at 
org.apache.solr.update.UpdateLog.getRecentUpdates(UpdateLog.java:1079)
        at 
org.apache.solr.update.UpdateLog.seedBucketsWithHighestVersion(UpdateLog.java:1579)
        at 
org.apache.solr.update.UpdateLog.seedBucketsWithHighestVersion(UpdateLog.java:1610)
        at org.apache.solr.core.SolrCore.seedVersionBuckets(SolrCore.java:877)
        at 
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:526)
        at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
Caused by: java.io.FileNotFoundException: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationHandlerTest_AD79661DF28559C5-001/jetty-002/cores/source_collection_shard1_replica2/data/tlog/tlog.0000000000000000008.1513034951642054656
 (No such file or directory)
        at java.io.RandomAccessFile.open0(Native Method)
        at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
        at 
org.apache.solr.update.CdcrTransactionLog.reopenOutputStream(CdcrTransactionLog.java:236)
        ... 7 more


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.CollectionsAPIDistributedZkTest

Error Message:
544 threads leaked from SUITE scope at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest:     1) Thread[id=15756, 
name=qtp303629789-15756, state=WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
         at 
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
         at com.sun.jmx.mbeanserver.Repository.retrieve(Repository.java:487)    
     at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1088)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:643)
         at 
com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)    
     at 
javax.management.AttributeValueExp.getAttribute(AttributeValueExp.java:163)     
    at javax.management.AttributeValueExp.apply(AttributeValueExp.java:105)     
    at javax.management.BinaryRelQueryExp.apply(BinaryRelQueryExp.java:109)     
    at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.filterListOfObjectInstances(DefaultMBeanServerInterceptor.java:1626)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.queryNames(DefaultMBeanServerInterceptor.java:546)
         at 
com.sun.jmx.mbeanserver.JmxMBeanServer.queryNames(JmxMBeanServer.java:619)      
   at org.apache.solr.core.JmxMonitoredMap.clear(JmxMonitoredMap.java:144)      
   at org.apache.solr.core.SolrCore.close(SolrCore.java:1257)         at 
org.apache.solr.core.CoreContainer.unload(CoreContainer.java:908)         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleUnloadAction(CoreAdminHandler.java:713)
         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:222)
         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:192)
         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:151)
         at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:673)  
       at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:441)      
   at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:214)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)    
     at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)     
    at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)      
   at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)       
  at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)   
      at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)  
       at org.eclipse.jetty.server.Server.handle(Server.java:499)         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)     
    at 
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)      
   at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) 
        at java.lang.Thread.run(Thread.java:745)    2) Thread[id=13801, 
name=searcherExecutor-4927-thread-1, state=WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    3) Thread[id=18262, 
name=zkCallback-1585-thread-84, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    4) Thread[id=15151, 
name=searcherExecutor-6063-thread-1, state=WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    5) Thread[id=18575, 
name=qtp2041769118-18575, state=BLOCKED, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleWaitForStateAction(CoreAdminHandler.java:1060)
         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:258)
         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:192)
         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:151)
         at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:673)  
       at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:441)      
   at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:214)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)    
     at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364)     
    at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)      
   at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)       
  at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)   
      at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)  
       at org.eclipse.jetty.server.Server.handle(Server.java:499)         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)     
    at 
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)      
   at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) 
        at java.lang.Thread.run(Thread.java:745)    6) Thread[id=13735, 
name=searcherExecutor-4870-thread-1, state=WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    7) Thread[id=18564, 
name=RecoveryThread-awholynewcollection_2_shard2_replica1, state=WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)         at 
java.util.concurrent.FutureTask.get(FutureTask.java:191)         at 
org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:598)
         at 
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:361)    
     at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)   
 8) Thread[id=18229, name=zkCallback-1585-thread-78, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    9) Thread[id=13557, 
name=Scheduler-397135422, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   10) Thread[id=16000, 
name=coreZkRegister-4748-thread-5-processing-n:127.0.0.1:60283_s_ 
x:awholynewstresscollection_collection3_3_shard4_replica2 s:shard4 
c:awholynewstresscollection_collection3_3 r:core_node32, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1071)       
  at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1041)  
       at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:1024)  
       at org.apache.solr.cloud.ZkController.register(ZkController.java:938)    
     at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:233)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   11) Thread[id=18139, 
name=zkCallback-1589-thread-37, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   12) Thread[id=16815, 
name=zkCallback-1585-thread-9-processing-n:127.0.0.1:36729_s_-EventThread, 
state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1071)       
  at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1041)  
       at 
org.apache.solr.cloud.ZkController.waitForLeaderToSeeDownState(ZkController.java:1632)
         at 
org.apache.solr.cloud.ZkController.registerAllCoresAsDown(ZkController.java:473)
         at 
org.apache.solr.cloud.ZkController.access$100(ZkController.java:119)         at 
org.apache.solr.cloud.ZkController$1.command(ZkController.java:302)         at 
org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:158)
         at 
org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57)
         at 
org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
         at 
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)   
      at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)   
13) Thread[id=15769, name=qtp303629789-15769, state=WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
         at 
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
         at com.sun.jmx.mbeanserver.Repository.retrieve(Repository.java:487)    
     at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1088)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:643)
         at 
com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)    
     at 
javax.management.AttributeValueExp.getAttribute(AttributeValueExp.java:163)     
    at javax.management.AttributeValueExp.apply(AttributeValueExp.java:105)     
    at javax.management.BinaryRelQueryExp.apply(BinaryRelQueryExp.java:109)     
    at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.filterListOfObjectInstances(DefaultMBeanServerInterceptor.java:1626)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.queryNames(DefaultMBeanServerInterceptor.java:546)
         at 
com.sun.jmx.mbeanserver.JmxMBeanServer.queryNames(JmxMBeanServer.java:619)      
   at org.apache.solr.core.JmxMonitoredMap.clear(JmxMonitoredMap.java:144)      
   at org.apache.solr.core.SolrCore.close(SolrCore.java:1257)         at 
org.apache.solr.core.CoreContainer.unload(CoreContainer.java:908)         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleUnloadAction(CoreAdminHandler.java:713)
         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:222)
         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:192)
         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:151)
         at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:673)  
       at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:441)      
   at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:214)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)    
     at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)     
    at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)      
   at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)       
  at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)   
      at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)  
       at org.eclipse.jetty.server.Server.handle(Server.java:499)         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)     
    at 
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)      
   at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) 
        at java.lang.Thread.run(Thread.java:745)   14) Thread[id=18509, 
name=RecoveryThread-awholynewcollection_1_shard5_replica1, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:511)    
     at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)   
15) Thread[id=18551, 
name=coreZkRegister-4748-thread-41-processing-n:127.0.0.1:60283_s_ 
x:awholynewstresscollection_collection3_3_shard4_replica2 s:shard4 
c:awholynewstresscollection_collection3_3 r:core_node32, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1071)       
  at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1041)  
       at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:997)   
      at org.apache.solr.cloud.ZkController.register(ZkController.java:938)     
    at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:233)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   16) Thread[id=14856, 
name=qtp2140027718-14856, state=WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
         at 
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
         at com.sun.jmx.mbeanserver.Repository.retrieve(Repository.java:487)    
     at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1088)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:643)
         at 
com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)    
     at 
javax.management.AttributeValueExp.getAttribute(AttributeValueExp.java:163)     
    at javax.management.AttributeValueExp.apply(AttributeValueExp.java:105)     
    at javax.management.BinaryRelQueryExp.apply(BinaryRelQueryExp.java:109)     
    at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.filterListOfObjectInstances(DefaultMBeanServerInterceptor.java:1626)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.queryNames(DefaultMBeanServerInterceptor.java:546)
         at 
com.sun.jmx.mbeanserver.JmxMBeanServer.queryNames(JmxMBeanServer.java:619)      
   at org.apache.solr.core.JmxMonitoredMap.clear(JmxMonitoredMap.java:144)      
   at org.apache.solr.core.SolrCore.close(SolrCore.java:1257)         at 
org.apache.solr.core.CoreContainer.unload(CoreContainer.java:908)         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleUnloadAction(CoreAdminHandler.java:713)
         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:222)
         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:192)
         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:151)
         at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:673)  
       at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:441)      
   at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:214)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)    
     at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)     
    at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)      
   at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)       
  at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)   
      at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)  
       at org.eclipse.jetty.server.Server.handle(Server.java:499)         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)     
    at 
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)      
   at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) 
        at java.lang.Thread.run(Thread.java:745)   17) Thread[id=15080, 
name=searcherExecutor-5937-thread-1, state=WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   18) Thread[id=18560, 
name=coreZkRegister-4748-thread-46-processing-n:127.0.0.1:60283_s_ 
x:awholynewstresscollection_collection3_3_shard4_replica2 s:shard4 
c:awholynewstresscollection_collection3_3 r:core_node32, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1071)       
  at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1041)  
       at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:997)   
      at org.apache.solr.cloud.ZkController.register(ZkController.java:938)     
    at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:233)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   19) Thread[id=15794, 
name=qtp2140027718-15794, state=WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
         at 
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
         at com.sun.jmx.mbeanserver.Repository.retrieve(Repository.java:487)    
     at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1088)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:643)
         at 
com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)    
     at 
javax.management.AttributeValueExp.getAttribute(AttributeValueExp.java:163)     
    at javax.management.AttributeValueExp.apply(AttributeValueExp.java:105)     
    at javax.management.BinaryRelQueryExp.apply(BinaryRelQueryExp.java:109)     
    at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.filterListOfObjectInstances(DefaultMBeanServerInterceptor.java:1626)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.queryNames(DefaultMBeanServerInterceptor.java:546)
         at 
com.sun.jmx.mbeanserver.JmxMBeanServer.queryNames(JmxMBeanServer.java:619)      
   at org.apache.solr.core.JmxMonitoredMap.clear(JmxMonitoredMap.java:144)      
   at org.apache.solr.core.SolrCore.close(SolrCore.java:1257)         at 
org.apache.solr.core.CoreContainer.unload(CoreContainer.java:908)         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleUnloadAction(CoreAdminHandler.java:713)
         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:222)
         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:192)
         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:151)
         at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:673)  
       at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:441)      
   at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:214)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)    
     at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)     
    at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)      
   at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)       
  at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)   
      at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)  
       at org.eclipse.jetty.server.Server.handle(Server.java:499)         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)     
    at 
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)      
   at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) 
        at java.lang.Thread.run(Thread.java:745)   20) Thread[id=17777, 
name=coreZkRegister-4726-thread-37-processing-n:127.0.0.1:36729_s_ 
x:awholynewstresscollection_collection3_3_shard7_replica4 s:shard7 
c:awholynewstresscollection_collection3_3 r:core_node27, state=WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1071)       
  at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1041)  
       at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:997)   
      at org.apache.solr.cloud.ZkController.register(ZkController.java:938)     
    at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:233)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   21) Thread[id=18522, 
name=coreZkRegister-4726-thread-64-processing-n:127.0.0.1:36729_s_ 
x:awholynewstresscollection_collection3_3_shard7_replica4 s:shard7 
c:awholynewstresscollection_collection3_3 r:core_node27, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:635)
         at 
org.apache.solr.common.cloud.ZkStateReader.getLeaderUrl(ZkStateReader.java:604) 
        at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:1022) 
        at org.apache.solr.cloud.ZkController.register(ZkController.java:938)   
      at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:233)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   22) Thread[id=18034, 
name=qtp1957971284-18034, state=BLOCKED, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleWaitForStateAction(CoreAdminHandler.java:1060)
         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:258)
         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:192)
         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:151)
         at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:673)  
       at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:441)      
   at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:214)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)    
     at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364)     
    at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)      
   at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)       
  at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)   
      at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)  
       at org.eclipse.jetty.server.Server.handle(Server.java:499)         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)     
    at 
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)      
   at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) 
        at java.lang.Thread.run(Thread.java:745)   23) Thread[id=13529, 
name=qtp1957971284-13529-selector-ServerConnectorManager@553dda4/0, 
state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at 
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at 
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)         at 
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at 
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at 
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at 
org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600)
         at 
org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549)
         at 
org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)  
       at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) 
        at java.lang.Thread.run(Thread.java:745)   24) Thread[id=17714, 
name=coreZkRegister-4737-thread-32-processing-n:127.0.0.1:38158_s_ 
x:awholynewstresscollection_collection3_3_shard5_replica1 s:shard5 
c:awholynewstresscollection_collection3_3 r:core_node31, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1071)       
  at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1041)  
       at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:997)   
      at org.apache.solr.cloud.ZkController.register(ZkController.java:938)     
    at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:233)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   25) Thread[id=17787, 
name=qtp303629789-17787, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) 
        at java.lang.Thread.run(Thread.java:745)   26) Thread[id=18442, 
name=zkCallback-1585-thread-101, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   27) Thread[id=18018, 
name=coreZkRegister-4726-thread-51-processing-n:127.0.0.1:36729_s_ 
x:awholynewstresscollection_collection3_3_shard7_replica4 s:shard7 
c:awholynewstresscollection_collection3_3 r:core_node27, state=WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1071)       
  at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1041)  
       at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:997)   
      at org.apache.solr.cloud.ZkController.register(ZkController.java:938)     
    at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:233)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   28) Thread[id=16885, 
name=zkCallback-1598-thread-12-processing-n:127.0.0.1:54030_s_-EventThread, 
state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1071)       
  at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1041)  
       at 
org.apache.solr.cloud.ZkController.waitForLeaderToSeeDownState(ZkController.java:1632)
         at 
org.apache.solr.cloud.ZkController.registerAllCoresAsDown(ZkController.java:473)
         at 
org.apache.solr.cloud.ZkController.access$100(ZkController.java:119)         at 
org.apache.solr.cloud.ZkController$1.command(ZkController.java:302)         at 
org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:158)
         at 
org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57)
         at 
org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
         at 
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)   
      at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)   
29) Thread[id=13567, name=Scheduler-512361265, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   30) Thread[id=18549, 
name=coreZkRegister-4748-thread-39-processing-n:127.0.0.1:60283_s_ 
x:awholynewstresscollection_collection3_3_shard4_replica2 s:shard4 
c:awholynewstresscollection_collection3_3 r:core_node32, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:527)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:291)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:197) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:346)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1113)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:926)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:233)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   31) Thread[id=13446, 
name=Scheduler-1946602227, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   32) Thread[id=16328, 
name=zkCallback-1593-thread-12-processing-n:127.0.0.1:60283_s_-EventThread, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1071)       
  at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1041)  
       at 
org.apache.solr.cloud.ZkController.waitForLeaderToSeeDownState(ZkController.java:1632)
         at 
org.apache.solr.cloud.ZkController.registerAllCoresAsDown(ZkController.java:473)
         at 
org.apache.solr.cloud.ZkController.access$100(ZkController.java:119)         at 
org.apache.solr.cloud.ZkController$1.command(ZkController.java:302)         at 
org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:158)
         at 
org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57)
         at 
org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
         at 
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)   
      at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)   
33) Thread[id=17798, 
name=coreZkRegister-4726-thread-45-processing-n:127.0.0.1:36729_s_ 
x:awholynewstresscollection_collection3_3_shard7_replica4 s:shard7 
c:awholynewstresscollection_collection3_3 r:core_node27, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1071)       
  at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1041)  
       at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:997)   
      at org.apache.solr.cloud.ZkController.register(ZkController.java:938)     
    at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:233)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   34) Thread[id=13799, 
name=searcherExecutor-4925-thread-1, state=WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   35) Thread[id=18397, 
name=zkCallback-1585-thread-97, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   36) Thread[id=18059, 
name=zkCallback-1589-thread-35, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   37) Thread[id=17958, 
name=zkCallback-1593-thread-35-processing-n:127.0.0.1:60283_s_, state=BLOCKED, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   38) Thread[id=15132, 
name=searcherExecutor-6074-thread-1, state=WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   39) Thread[id=17691, 
name=coreZkRegister-4748-thread-26-processing-n:127.0.0.1:60283_s_ 
x:awholynewstresscollection_collection3_3_shard4_replica2 s:shard4 
c:awholynewstresscollection_collection3_3 r:core_node32, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1071)       
  at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1041)  
       at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:1024)  
       at org.apache.solr.cloud.ZkController.register(ZkController.java:938)    
     at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:233)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   40) Thread[id=17201, 
name=zkCallback-1598-thread-12-processing-n:127.0.0.1:54030_s_-EventThread, 
state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.net.SocketInputStream.socketRead0(Native Method)         at 
java.net.SocketInputStream.socketRead(SocketInputStream.java:116)         at 
java.net.SocketInputStream.read(SocketInputStream.java:170)         at 
java.net.SocketInputStream.read(SocketInputStream.java:141)         at 
org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
         at 
org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84) 
        at 
org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
         at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
         at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
         at 
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
         at 
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
         at 
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
         at 
org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
         at 
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
         at 
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
         at 
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
         at 
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
         at 
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
         at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
         at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
         at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
         at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:480)
         at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
         at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
         at 
org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)         
at 
org.apache.solr.cloud.ZkController.waitForLeaderToSeeDownState(ZkController.java:1693)
         at 
org.apache.solr.cloud.ZkController.registerAllCoresAsDown(ZkController.java:473)
         at 
org.apache.solr.cloud.ZkController.access$100(ZkController.java:119)         at 
org.apache.solr.cloud.ZkController$1.command(ZkController.java:302)         at 
org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:158)
         at 
org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57)
         at 
org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
         at 
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)   
      at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)   
41) Thread[id=18542, 
name=coreZkRegister-4748-thread-33-processing-n:127.0.0.1:60283_s_ 
x:awholynewstresscollection_collection3_3_shard4_replica2 s:shard4 
c:awholynewstresscollection_collection3_3 r:core_node32, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ZkController.getLeader(ZkController.java:1021)         at 
org.apache.solr.cloud.ZkController.register(ZkController.java:938)         at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:233)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   42) Thread[id=17251, 
name=zkCallback-1585-thread-9-processing-n:127.0.0.1:36729_s_-EventThread, 
state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)   
43) Thread[id=18541, name=coreZkRegister-4748-thread-32, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   44) Thread[id=18093, 
name=qtp303629789-18093, state=BLOCKED, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleWaitForStateAction(CoreAdminHandler.java:1060)
         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:258)
         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:192)
         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:151)
         at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:673)  
       at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:441)      
   at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:214)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)    
     at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364)     
    at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)      
   at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)       
  at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)   
      at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)  
       at org.eclipse.jetty.server.Server.handle(Server.java:499)         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)     
    at 
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)      
   at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) 
        at java.lang.Thread.run(Thread.java:745)   45) Thread[id=18486, 
name=coreZkRegister-4737-thread-34, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   46) Thread[id=15682, 
name=qtp1398700481-15682, state=WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
         at 
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
         at com.sun.jmx.mbeanserver.Repository.retrieve(Repository.java:487)    
     at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1088)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:643)
         at 
com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)    
     at 
javax.management.AttributeValueExp.getAttribute(AttributeValueExp.java:163)     
    at javax.management.AttributeValueExp.apply(AttributeValueExp.java:105)     
    at javax.management.BinaryRelQueryExp.apply(BinaryRelQueryExp.java:109)     
    at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.filterListOfObjectInstances(DefaultMBeanServerInterceptor.java:1626)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.queryNames(DefaultMBeanServerInterceptor.java:546)
         at 
com.sun.jmx.mbeanserver.JmxMBeanServer.queryNames(JmxMBeanServer.java:619)      
   at org.apache.solr.core.JmxMonitoredMap.clear(JmxMonitoredMap.java:144)      
   at org.apache.solr.core.SolrCore.close(SolrCore.java:1257)         at 
org.apache.solr.core.CoreContainer.unload(CoreContainer.java:908)         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleUnloadAction(CoreAdminHandler.java:713)
         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:222)
         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:192)
         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:151)
         at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:673)  
       at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:441)      
   at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:214)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)    
     at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)     
    at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)      
   at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)       
  at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)   
      at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)  
       at org.eclipse.jetty.server.Server.handle(Server.java:499)         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)     
    at 
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)      
   at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) 
        at java.lang.Thread.run(Thread.java:745)   47) Thread[id=15784, 
name=qtp2140027718-15784, state=WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
         at 
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
         at com.sun.jmx.mbeanserver.Repository.retrieve(Repository.java:487)    
     at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1088)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:643)
         at 
com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)    
     at 
javax.management.AttributeValueExp.getAttribute(AttributeValueExp.java:163)     
    at javax.management.AttributeValueExp.apply(AttributeValueExp.java:105)     
    at javax.management.BinaryRelQueryExp.apply(BinaryRelQueryExp.java:109)     
    at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.filterListOfObjectInstances(DefaultMBeanServerInterceptor.java:1626)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.queryNames(DefaultMBeanServerInterceptor.java:546)
         at 
com.sun.jmx.mbeanserver.JmxMBeanServer.queryNames(JmxMBeanServer.java:619)      
   at org.apache.solr.core.JmxMonitoredMap.clear(JmxMonitoredMap.java:144)      
   at org.apache.solr.core.SolrCore.close(SolrCore.java:1257)         at 
org.apache.solr.core.CoreContainer.unload(CoreContainer.java:908)         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleUnloadAction(CoreAdminHandler.java:713)
         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:222)
         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:192)
         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:151)
         at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:673)  
       at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:441)      
   at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:214)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)    
     at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)     
    at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)      
   at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)       
  at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)   
      at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)  
       at org.eclipse.jetty.server.Server.handle(Server.java:499)         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)     
    at 
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)      
   at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) 
        at java.lang.Thread.run(Thread.java:745)   48) Thread[id=15979, 
name=coreZkRegister-4726-thread-16-processing-n:127.0.0.1:36729_s_ 
x:awholynewstresscollection_collection3_3_shard7_replica4 s:shard7 
c:awholynewstresscollection_collection3_3 r:core_node27, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1071)       
  at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1041)  
       at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:1024)  
       at org.apache.solr.cloud.ZkController.register(ZkController.java:938)    
     at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:233)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   49) Thread[id=18243, 
name=qtp2041769118-18243, state=BLOCKED, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleWaitForStateAction(CoreAdminHandler.java:1060)
         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:258)
         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:192)
         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:151)
         at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:673)  
       at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:441)      
   at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:214)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)    
     at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364)     
    at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)      
   at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)       
  at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)   
      at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)  
       at org.eclipse.jetty.server.Server.handle(Server.java:499)         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)     
    at 
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)      
   at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) 
        at java.lang.Thread.run(Thread.java:745)   50) Thread[id=18240, 
name=qtp2041769118-18240, state=BLOCKED, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleWaitForStateAction(CoreAdminHandler.java:1060)
         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:258)
         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:192)
         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:151)
         at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:673)  
       at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:441)      
   at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:214)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)    
     at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364)     
    at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)      
   at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)       
  at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)   
      at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)  
       at org.eclipse.jetty.server.Server.handle(Server.java:499)         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)     
    at 
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)      
   at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) 
        at java.lang.Thread.run(Thread.java:745)   51) Thread[id=14847, 
name=qtp2140027718-14847, state=WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
         at 
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
         at com.sun.jmx.mbeanserver.Repository.retrieve(Repository.java:487)    
     at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1088)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:643)
         at 
com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)    
     at 
javax.management.AttributeValueExp.getAttribute(AttributeValueExp.java:163)     
    at javax.management.AttributeValueExp.apply(AttributeValueExp.java:105)     
    at javax.management.BinaryRelQueryExp.apply(BinaryRelQueryExp.java:109)     
    at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.filterListOfObjectInstances(DefaultMBeanServerInterceptor.java:1626)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.queryNames(DefaultMBeanServerInterceptor.java:546)
         at 
com.sun.jmx.mbeanserver.JmxMBeanServer.queryNames(JmxMBeanServer.java:619)      
   at org.apache.solr.core.JmxMonitoredMap.clear(JmxMonitoredMap.java:144)      
   at org.apache.solr.core.SolrCore.close(SolrCore.java:1257)         at 
org.apache.solr.core.CoreContainer.unload(CoreContainer.java:908)         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleUnloadAction(CoreAdminHandler.java:713)
         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:222)
         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:192)
         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:151)
         at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:673)  
       at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:441)      
   at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:214)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)    
     at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)     
    at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)      
   at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)       
  at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)   
      at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)  
       at org.eclipse.jetty.server.Server.handle(Server.java:499)         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)     
    at 
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)      
   at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) 
        at java.lang.Thread.run(Thread.java:745)   52) Thread[id=18023, 
name=coreZkRegister-4726-thread-56-processing-n:127.0.0.1:36729_s_ 
x:awholynewstresscollection_collection3_3_shard7_replica4 s:shard7 
c:awholynewstresscollection_collection3_3 r:core_node27, state=RUNNABLE, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1071)       
  at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1041)  
       at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:997)   
      at org.apache.solr.cloud.ZkController.register(ZkController.java:938)     
    at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:233)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   53) Thread[id=18065, 
name=qtp303629789-18065, state=BLOCKED, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleWaitForStateAction(CoreAdminHandler.java:1060)
         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:258)
         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:192)
         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:151)
         at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:673)  
       at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:441)      
   at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:214)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)
         at org.eclipse.jetty.se

[...truncated too long message...]

.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   419) Thread[id=17710, name=coreZkRegister-4737-thread-28, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   420) Thread[id=17712, name=coreZkRegister-4737-thread-30, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   421) Thread[id=17677, name=zkCallback-1593-thread-25, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   422) Thread[id=18454, name=zkCallback-1585-thread-102, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   423) Thread[id=18003, name=zkCallback-1585-thread-56, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   424) Thread[id=18005, name=zkCallback-1585-thread-58, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   425) Thread[id=15058, name=searcherExecutor-5901-thread-1, 
state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([AD79661DF28559C5]:0)
   [junit4] Completed [466/546] on J1 in 2868.82s, 1 test, 3 errors <<< 
FAILURES!

[...truncated 234 lines...]
   [junit4] JVM J1: stdout was not empty, see: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/temp/junit4-J1-20150922_125441_031.sysout
   [junit4] >>> JVM J1: stdout (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Dumping heap to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/heapdumps/java_pid16717.hprof
 ...
   [junit4] Heap dump file created [668548476 bytes in 4.380 secs]
   [junit4] <<< JVM J1: EOF ----

[...truncated 21 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:782:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:719:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:59:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:233:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:514:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1432:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:987:
 There were test failures: 546 suites (8 ignored), 2145 tests, 2 suite-level 
errors, 3 errors, 2 failures, 317 ignored (17 assumptions) [seed: 
AD79661DF28559C5]

Total time: 225 minutes 48 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org