[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 7 - Still Failing

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/7/

6 tests failed.
FAILED:  org.apache.lucene.index.TestIndexSorting.testRandom3

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([33668C7C0492929E]:0)


FAILED:  junit.framework.TestSuite.org.apache.lucene.index.TestIndexSorting

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([33668C7C0492929E]:0)


FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:3332)
        at 
java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
        at 
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:649)
        at java.lang.StringBuilder.append(StringBuilder.java:202)
        at 
org.apache.http.client.utils.URLEncodedUtils.urlEncode(URLEncodedUtils.java:522)
        at 
org.apache.http.client.utils.URLEncodedUtils.encodeFormFields(URLEncodedUtils.java:638)
        at 
org.apache.http.client.utils.URLEncodedUtils.format(URLEncodedUtils.java:390)
        at 
org.apache.http.client.utils.URLEncodedUtils.format(URLEncodedUtils.java:369)
        at 
org.apache.http.client.entity.UrlEncodedFormEntity.<init>(UrlEncodedFormEntity.java:77)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:447)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1121)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:862)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:974)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:990)
        at 
org.apache.solr.cloud.CloudInspectUtil.compareResults(CloudInspectUtil.java:228)
        at 
org.apache.solr.cloud.CloudInspectUtil.compareResults(CloudInspectUtil.java:167)
        at 
org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexingBatchPerRequestWithHttpSolrClient(FullSolrCloudDistribCmdsTest.java:668)
        at 
org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test(FullSolrCloudDistribCmdsTest.java:152)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Error Message:
20 threads leaked from SUITE scope at 
org.apache.solr.cloud.FullSolrCloudDistribCmdsTest:     1) Thread[id=20390, 
name=searcherExecutor-5760-thread-1, state=WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)    2) Thread[id=20080, 
name=qtp1812211277-20080, state=BLOCKED, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at 
org.apache.log4j.Category.callAppenders(Category.java:204)         at 
org.apache.log4j.Category.forcedLog(Category.java:391)         at 
org.apache.log4j.Category.log(Category.java:856)         at 
org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:304)         at 
org.apache.solr.core.SolrCore.execute(SolrCore.java:2478)         at 
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)         at 
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)      
   at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)       
  at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)   
      at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)  
       at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) 
        at org.eclipse.jetty.server.Server.handle(Server.java:534)         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)     
    at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)    
     at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) 
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) 
        at java.lang.Thread.run(Thread.java:748)    3) Thread[id=20402, 
name=searcherExecutor-5763-thread-1, state=WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)    4) Thread[id=20389, 
name=searcherExecutor-5761-thread-1, state=WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)    5) Thread[id=20111, 
name=qtp150412892-20111, state=BLOCKED, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at 
org.apache.lucene.store.MockIndexInputWrapper.readByte(MockIndexInputWrapper.java:140)
         at org.apache.lucene.codecs.compressing.LZ4.decompress(LZ4.java:110)   
      at 
org.apache.lucene.codecs.compressing.CompressionMode$4.decompress(CompressionMode.java:138)
         at 
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader$BlockState.document(CompressingStoredFieldsReader.java:555)
         at 
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.document(CompressingStoredFieldsReader.java:571)
         at 
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:578)
         at 
org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat$AssertingStoredFieldsReader.visitDocument(AssertingStoredFieldsFormat.java:73)
         at org.apache.lucene.index.CodecReader.document(CodecReader.java:83)   
      at 
org.apache.lucene.index.FilterLeafReader.document(FilterLeafReader.java:341)    
     at 
org.apache.lucene.index.FilterLeafReader.document(FilterLeafReader.java:341)    
     at 
org.apache.lucene.index.BaseCompositeReader.document(BaseCompositeReader.java:118)
         at 
org.apache.solr.search.SolrDocumentFetcher.doc(SolrDocumentFetcher.java:198)    
     at org.apache.solr.response.DocsStreamer.next(DocsStreamer.java:155)       
  at org.apache.solr.response.DocsStreamer.next(DocsStreamer.java:59)         
at 
org.apache.solr.response.BinaryResponseWriter$Resolver.writeResultsBody(BinaryResponseWriter.java:126)
         at 
org.apache.solr.response.BinaryResponseWriter$Resolver.writeResults(BinaryResponseWriter.java:145)
         at 
org.apache.solr.response.BinaryResponseWriter$Resolver.resolve(BinaryResponseWriter.java:89)
         at 
org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:239)        
 at 
org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)  
       at 
org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)  
       at 
org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)        
 at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)     
    at 
org.apache.solr.response.BinaryResponseWriter.write(BinaryResponseWriter.java:52)
         at 
org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:49)
         at 
org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:806)       
  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:535)         
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)      
   at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)       
  at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)   
      at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)  
       at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) 
        at org.eclipse.jetty.server.Server.handle(Server.java:534)         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)     
    at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)    
     at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) 
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) 
        at java.lang.Thread.run(Thread.java:748)    6) Thread[id=20398, 
name=searcherExecutor-5765-thread-1, state=WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)    7) Thread[id=20525, 
name=qtp150412892-20525, state=RUNNABLE, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at 
java.io.FilterOutputStream.write(FilterOutputStream.java:125)         at 
org.apache.lucene.util.TestRuleLimitSysouts$DelegateStream.write(TestRuleLimitSysouts.java:128)
         at java.io.PrintStream.write(PrintStream.java:480)         at 
sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)         at 
sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)         at 
sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)         at 
sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)         at 
java.io.OutputStreamWriter.write(OutputStreamWriter.java:220)         at 
java.io.Writer.write(Writer.java:157)         at 
org.apache.log4j.helpers.QuietWriter.write(QuietWriter.java:48)         at 
org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:310)         at 
org.apache.log4j.WriterAppender.append(WriterAppender.java:162)         at 
org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)         
at 
org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
         at org.apache.log4j.Category.callAppenders(Category.java:206)         
at org.apache.log4j.Category.forcedLog(Category.java:391)         at 
org.apache.log4j.Category.log(Category.java:856)         at 
org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:304)         at 
org.apache.solr.core.SolrCore.execute(SolrCore.java:2478)         at 
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)         at 
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)      
   at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)       
  at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)   
      at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)  
       at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) 
        at org.eclipse.jetty.server.Server.handle(Server.java:534)         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)     
    at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)    
     at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) 
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) 
        at java.lang.Thread.run(Thread.java:748)    8) Thread[id=20514, 
name=qtp423659769-20514, state=BLOCKED, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at 
org.apache.log4j.Category.callAppenders(Category.java:204)         at 
org.apache.log4j.Category.forcedLog(Category.java:391)         at 
org.apache.log4j.Category.log(Category.java:856)         at 
org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:304)         at 
org.apache.solr.core.SolrCore.execute(SolrCore.java:2478)         at 
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)         at 
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)      
   at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)       
  at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)   
      at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)  
       at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) 
        at org.eclipse.jetty.server.Server.handle(Server.java:534)         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)     
    at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)    
     at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) 
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) 
        at java.lang.Thread.run(Thread.java:748)    9) Thread[id=20489, 
name=qtp423659769-20489, state=BLOCKED, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at 
org.apache.log4j.Category.callAppenders(Category.java:204)         at 
org.apache.log4j.Category.forcedLog(Category.java:391)         at 
org.apache.log4j.Category.log(Category.java:856)         at 
org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:381)         at 
org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:810)       
  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:535)         
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)      
   at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)       
  at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)   
      at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)  
       at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) 
        at org.eclipse.jetty.server.Server.handle(Server.java:534)         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)     
    at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)    
     at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) 
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) 
        at java.lang.Thread.run(Thread.java:748)   10) Thread[id=20501, 
name=qtp1812211277-20501, state=RUNNABLE, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at 
java.util.Arrays.copyOfRange(Arrays.java:3664)         at 
java.lang.String.<init>(String.java:207)         at 
java.lang.StringBuilder.toString(StringBuilder.java:407)         at 
org.apache.lucene.store.MockIndexInputWrapper.<init>(MockIndexInputWrapper.java:40)
         at 
org.apache.lucene.store.MockIndexInputWrapper.clone(MockIndexInputWrapper.java:89)
         at 
org.apache.lucene.store.MockIndexInputWrapper.clone(MockIndexInputWrapper.java:29)
         at 
org.apache.lucene.codecs.blockterms.BlockTermsReader$FieldReader$SegmentTermsEnum.<init>(BlockTermsReader.java:331)
         at 
org.apache.lucene.codecs.blockterms.BlockTermsReader$FieldReader.iterator(BlockTermsReader.java:243)
         at 
org.apache.lucene.search.TermQuery$TermWeight.getTermsEnum(TermQuery.java:121)  
       at 
org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:90)         
at org.apache.lucene.search.Weight.bulkScorer(Weight.java:147)         at 
org.apache.lucene.search.BooleanWeight.optionalBulkScorer(BooleanWeight.java:157)
         at 
org.apache.lucene.search.BooleanWeight.booleanScorer(BooleanWeight.java:214)    
     at 
org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:283)       
  at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:655)      
   at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)     
    at 
org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:215)
         at 
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1591)
         at 
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1408)
         at 
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:575)     
    at 
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:544)
         at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2473)         
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)         
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)      
   at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)       
  at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)   
      at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)  
       at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) 
        at org.eclipse.jetty.server.Server.handle(Server.java:534)         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)     
    at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)    
     at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) 
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) 
        at java.lang.Thread.run(Thread.java:748)   11) Thread[id=20400, 
name=searcherExecutor-5773-thread-1, state=WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)   12) Thread[id=20401, 
name=searcherExecutor-5767-thread-1, state=WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)   13) Thread[id=20393, 
name=searcherExecutor-5769-thread-1, state=WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)   14) Thread[id=20410, 
name=searcherExecutor-5771-thread-1, state=WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)   15) Thread[id=20166, 
name=qtp2106581206-20166, state=RUNNABLE, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at 
java.lang.String.valueOf(String.java:2994)         at 
java.lang.StringBuilder.append(StringBuilder.java:131)         at 
org.apache.lucene.store.MockIndexInputWrapper.<init>(MockIndexInputWrapper.java:40)
         at 
org.apache.lucene.store.MockIndexInputWrapper.clone(MockIndexInputWrapper.java:89)
         at 
org.apache.lucene.store.MockIndexInputWrapper.clone(MockIndexInputWrapper.java:29)
         at 
org.apache.lucene.codecs.blockterms.BlockTermsReader$FieldReader$SegmentTermsEnum.<init>(BlockTermsReader.java:331)
         at 
org.apache.lucene.codecs.blockterms.BlockTermsReader$FieldReader.iterator(BlockTermsReader.java:243)
         at 
org.apache.lucene.search.TermQuery$TermWeight.getTermsEnum(TermQuery.java:121)  
       at 
org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:90)         
at org.apache.lucene.search.Weight.bulkScorer(Weight.java:147)         at 
org.apache.lucene.search.BooleanWeight.optionalBulkScorer(BooleanWeight.java:157)
         at 
org.apache.lucene.search.BooleanWeight.booleanScorer(BooleanWeight.java:214)    
     at 
org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:283)       
  at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:655)      
   at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)     
    at 
org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:215)
         at 
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1591)
         at 
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1408)
         at 
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:575)     
    at 
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:544)
         at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2473)         
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)         
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)      
   at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)       
  at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)   
      at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)  
       at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) 
        at org.eclipse.jetty.server.Server.handle(Server.java:534)         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)     
    at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)    
     at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) 
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) 
        at java.lang.Thread.run(Thread.java:748)   16) Thread[id=20408, 
name=searcherExecutor-5776-thread-1, state=WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)   17) Thread[id=20543, 
name=qtp1812211277-20543, state=RUNNABLE, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at 
org.apache.lucene.codecs.blockterms.VariableGapTermsIndexReader$IndexEnum.<init>(VariableGapTermsIndexReader.java:97)
         at 
org.apache.lucene.codecs.blockterms.VariableGapTermsIndexReader.getFieldEnum(VariableGapTermsIndexReader.java:189)
         at 
org.apache.lucene.codecs.blockterms.BlockTermsReader$FieldReader$SegmentTermsEnum.<init>(BlockTermsReader.java:333)
         at 
org.apache.lucene.codecs.blockterms.BlockTermsReader$FieldReader.iterator(BlockTermsReader.java:243)
         at 
org.apache.lucene.search.TermQuery$TermWeight.getTermsEnum(TermQuery.java:121)  
       at 
org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:90)         
at org.apache.lucene.search.Weight.bulkScorer(Weight.java:147)         at 
org.apache.lucene.search.BooleanWeight.optionalBulkScorer(BooleanWeight.java:157)
         at 
org.apache.lucene.search.BooleanWeight.booleanScorer(BooleanWeight.java:214)    
     at 
org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:283)       
  at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:655)      
   at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)     
    at 
org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:215)
         at 
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1591)
         at 
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1408)
         at 
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:575)     
    at 
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:544)
         at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2473)         
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)         
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)      
   at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)       
  at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)   
      at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)  
       at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) 
        at org.eclipse.jetty.server.Server.handle(Server.java:534)         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)     
    at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)    
     at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) 
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) 
        at java.lang.Thread.run(Thread.java:748)   18) Thread[id=20397, 
name=searcherExecutor-5781-thread-1, state=WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)   19) Thread[id=20495, 
name=qtp423659769-20495, state=BLOCKED, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at 
org.apache.log4j.Category.callAppenders(Category.java:204)         at 
org.apache.log4j.Category.forcedLog(Category.java:391)         at 
org.apache.log4j.Category.log(Category.java:856)         at 
org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:304)         at 
org.apache.solr.core.SolrCore.execute(SolrCore.java:2478)         at 
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)         at 
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)      
   at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)       
  at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)   
      at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)  
       at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) 
        at org.eclipse.jetty.server.Server.handle(Server.java:534)         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)     
    at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)    
     at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) 
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) 
        at java.lang.Thread.run(Thread.java:748)   20) Thread[id=20512, 
name=qtp150412892-20512, state=BLOCKED, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at 
org.apache.log4j.Category.callAppenders(Category.java:204)         at 
org.apache.log4j.Category.forcedLog(Category.java:391)         at 
org.apache.log4j.Category.log(Category.java:856)         at 
org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:304)         at 
org.apache.solr.core.SolrCore.execute(SolrCore.java:2478)         at 
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)         at 
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)      
   at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)       
  at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)   
      at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)  
       at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) 
        at org.eclipse.jetty.server.Server.handle(Server.java:534)         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)     
    at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)    
     at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) 
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) 
        at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 20 threads leaked from 
SUITE scope at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest: 
   1) Thread[id=20390, name=searcherExecutor-5760-thread-1, state=WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=20080, name=qtp1812211277-20080, state=BLOCKED, 
group=TGRP-FullSolrCloudDistribCmdsTest]
        at org.apache.log4j.Category.callAppenders(Category.java:204)
        at org.apache.log4j.Category.forcedLog(Category.java:391)
        at org.apache.log4j.Category.log(Category.java:856)
        at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:304)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2478)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=20402, name=searcherExecutor-5763-thread-1, state=WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=20389, name=searcherExecutor-5761-thread-1, state=WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=20111, name=qtp150412892-20111, state=BLOCKED, 
group=TGRP-FullSolrCloudDistribCmdsTest]
        at 
org.apache.lucene.store.MockIndexInputWrapper.readByte(MockIndexInputWrapper.java:140)
        at org.apache.lucene.codecs.compressing.LZ4.decompress(LZ4.java:110)
        at 
org.apache.lucene.codecs.compressing.CompressionMode$4.decompress(CompressionMode.java:138)
        at 
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader$BlockState.document(CompressingStoredFieldsReader.java:555)
        at 
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.document(CompressingStoredFieldsReader.java:571)
        at 
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:578)
        at 
org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat$AssertingStoredFieldsReader.visitDocument(AssertingStoredFieldsFormat.java:73)
        at org.apache.lucene.index.CodecReader.document(CodecReader.java:83)
        at 
org.apache.lucene.index.FilterLeafReader.document(FilterLeafReader.java:341)
        at 
org.apache.lucene.index.FilterLeafReader.document(FilterLeafReader.java:341)
        at 
org.apache.lucene.index.BaseCompositeReader.document(BaseCompositeReader.java:118)
        at 
org.apache.solr.search.SolrDocumentFetcher.doc(SolrDocumentFetcher.java:198)
        at org.apache.solr.response.DocsStreamer.next(DocsStreamer.java:155)
        at org.apache.solr.response.DocsStreamer.next(DocsStreamer.java:59)
        at 
org.apache.solr.response.BinaryResponseWriter$Resolver.writeResultsBody(BinaryResponseWriter.java:126)
        at 
org.apache.solr.response.BinaryResponseWriter$Resolver.writeResults(BinaryResponseWriter.java:145)
        at 
org.apache.solr.response.BinaryResponseWriter$Resolver.resolve(BinaryResponseWriter.java:89)
        at 
org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:239)
        at 
org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)
        at 
org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)
        at 
org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at 
org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)
        at 
org.apache.solr.response.BinaryResponseWriter.write(BinaryResponseWriter.java:52)
        at 
org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:49)
        at 
org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:806)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:535)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=20398, name=searcherExecutor-5765-thread-1, state=WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   7) Thread[id=20525, name=qtp150412892-20525, state=RUNNABLE, 
group=TGRP-FullSolrCloudDistribCmdsTest]
        at java.io.FilterOutputStream.write(FilterOutputStream.java:125)
        at 
org.apache.lucene.util.TestRuleLimitSysouts$DelegateStream.write(TestRuleLimitSysouts.java:128)
        at java.io.PrintStream.write(PrintStream.java:480)
        at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
        at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
        at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
        at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)
        at java.io.OutputStreamWriter.write(OutputStreamWriter.java:220)
        at java.io.Writer.write(Writer.java:157)
        at org.apache.log4j.helpers.QuietWriter.write(QuietWriter.java:48)
        at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:310)
        at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)
        at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
        at 
org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
        at org.apache.log4j.Category.callAppenders(Category.java:206)
        at org.apache.log4j.Category.forcedLog(Category.java:391)
        at org.apache.log4j.Category.log(Category.java:856)
        at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:304)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2478)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)
   8) Thread[id=20514, name=qtp423659769-20514, state=BLOCKED, 
group=TGRP-FullSolrCloudDistribCmdsTest]
        at org.apache.log4j.Category.callAppenders(Category.java:204)
        at org.apache.log4j.Category.forcedLog(Category.java:391)
        at org.apache.log4j.Category.log(Category.java:856)
        at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:304)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2478)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)
   9) Thread[id=20489, name=qtp423659769-20489, state=BLOCKED, 
group=TGRP-FullSolrCloudDistribCmdsTest]
        at org.apache.log4j.Category.callAppenders(Category.java:204)
        at org.apache.log4j.Category.forcedLog(Category.java:391)
        at org.apache.log4j.Category.log(Category.java:856)
        at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:381)
        at 
org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:810)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:535)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)
  10) Thread[id=20501, name=qtp1812211277-20501, state=RUNNABLE, 
group=TGRP-FullSolrCloudDistribCmdsTest]
        at java.util.Arrays.copyOfRange(Arrays.java:3664)
        at java.lang.String.<init>(String.java:207)
        at java.lang.StringBuilder.toString(StringBuilder.java:407)
        at 
org.apache.lucene.store.MockIndexInputWrapper.<init>(MockIndexInputWrapper.java:40)
        at 
org.apache.lucene.store.MockIndexInputWrapper.clone(MockIndexInputWrapper.java:89)
        at 
org.apache.lucene.store.MockIndexInputWrapper.clone(MockIndexInputWrapper.java:29)
        at 
org.apache.lucene.codecs.blockterms.BlockTermsReader$FieldReader$SegmentTermsEnum.<init>(BlockTermsReader.java:331)
        at 
org.apache.lucene.codecs.blockterms.BlockTermsReader$FieldReader.iterator(BlockTermsReader.java:243)
        at 
org.apache.lucene.search.TermQuery$TermWeight.getTermsEnum(TermQuery.java:121)
        at 
org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:90)
        at org.apache.lucene.search.Weight.bulkScorer(Weight.java:147)
        at 
org.apache.lucene.search.BooleanWeight.optionalBulkScorer(BooleanWeight.java:157)
        at 
org.apache.lucene.search.BooleanWeight.booleanScorer(BooleanWeight.java:214)
        at 
org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:283)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:655)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)
        at 
org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:215)
        at 
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1591)
        at 
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1408)
        at 
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:575)
        at 
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:544)
        at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
        at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2473)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)
  11) Thread[id=20400, name=searcherExecutor-5773-thread-1, state=WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
  12) Thread[id=20401, name=searcherExecutor-5767-thread-1, state=WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
  13) Thread[id=20393, name=searcherExecutor-5769-thread-1, state=WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
  14) Thread[id=20410, name=searcherExecutor-5771-thread-1, state=WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
  15) Thread[id=20166, name=qtp2106581206-20166, state=RUNNABLE, 
group=TGRP-FullSolrCloudDistribCmdsTest]
        at java.lang.String.valueOf(String.java:2994)
        at java.lang.StringBuilder.append(StringBuilder.java:131)
        at 
org.apache.lucene.store.MockIndexInputWrapper.<init>(MockIndexInputWrapper.java:40)
        at 
org.apache.lucene.store.MockIndexInputWrapper.clone(MockIndexInputWrapper.java:89)
        at 
org.apache.lucene.store.MockIndexInputWrapper.clone(MockIndexInputWrapper.java:29)
        at 
org.apache.lucene.codecs.blockterms.BlockTermsReader$FieldReader$SegmentTermsEnum.<init>(BlockTermsReader.java:331)
        at 
org.apache.lucene.codecs.blockterms.BlockTermsReader$FieldReader.iterator(BlockTermsReader.java:243)
        at 
org.apache.lucene.search.TermQuery$TermWeight.getTermsEnum(TermQuery.java:121)
        at 
org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:90)
        at org.apache.lucene.search.Weight.bulkScorer(Weight.java:147)
        at 
org.apache.lucene.search.BooleanWeight.optionalBulkScorer(BooleanWeight.java:157)
        at 
org.apache.lucene.search.BooleanWeight.booleanScorer(BooleanWeight.java:214)
        at 
org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:283)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:655)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)
        at 
org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:215)
        at 
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1591)
        at 
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1408)
        at 
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:575)
        at 
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:544)
        at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
        at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2473)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)
  16) Thread[id=20408, name=searcherExecutor-5776-thread-1, state=WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
  17) Thread[id=20543, name=qtp1812211277-20543, state=RUNNABLE, 
group=TGRP-FullSolrCloudDistribCmdsTest]
        at 
org.apache.lucene.codecs.blockterms.VariableGapTermsIndexReader$IndexEnum.<init>(VariableGapTermsIndexReader.java:97)
        at 
org.apache.lucene.codecs.blockterms.VariableGapTermsIndexReader.getFieldEnum(VariableGapTermsIndexReader.java:189)
        at 
org.apache.lucene.codecs.blockterms.BlockTermsReader$FieldReader$SegmentTermsEnum.<init>(BlockTermsReader.java:333)
        at 
org.apache.lucene.codecs.blockterms.BlockTermsReader$FieldReader.iterator(BlockTermsReader.java:243)
        at 
org.apache.lucene.search.TermQuery$TermWeight.getTermsEnum(TermQuery.java:121)
        at 
org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:90)
        at org.apache.lucene.search.Weight.bulkScorer(Weight.java:147)
        at 
org.apache.lucene.search.BooleanWeight.optionalBulkScorer(BooleanWeight.java:157)
        at 
org.apache.lucene.search.BooleanWeight.booleanScorer(BooleanWeight.java:214)
        at 
org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:283)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:655)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)
        at 
org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:215)
        at 
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1591)
        at 
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1408)
        at 
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:575)
        at 
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:544)
        at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
        at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2473)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)
  18) Thread[id=20397, name=searcherExecutor-5781-thread-1, state=WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
  19) Thread[id=20495, name=qtp423659769-20495, state=BLOCKED, 
group=TGRP-FullSolrCloudDistribCmdsTest]
        at org.apache.log4j.Category.callAppenders(Category.java:204)
        at org.apache.log4j.Category.forcedLog(Category.java:391)
        at org.apache.log4j.Category.log(Category.java:856)
        at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:304)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2478)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java

[...truncated too long message...]

.handle(GzipHandler.java:395)
   [junit4]    >         at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]    >         at 
org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]    >         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]    >         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]    >         at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   [junit4]    >         at 
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   [junit4]    >         at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >   14) Thread[id=20495, name=qtp423659769-20495, state=BLOCKED, 
group=TGRP-FullSolrCloudDistribCmdsTest]
   [junit4]    >         at 
org.apache.log4j.Category.callAppenders(Category.java:204)
   [junit4]    >         at 
org.apache.log4j.Category.forcedLog(Category.java:391)
   [junit4]    >         at org.apache.log4j.Category.log(Category.java:856)
   [junit4]    >         at 
org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:381)
   [junit4]    >         at 
org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:810)
   [junit4]    >         at 
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:535)
   [junit4]    >         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
   [junit4]    >         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
   [junit4]    >         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]    >         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]    >         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]    >         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]    >         at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]    >         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]    >         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]    >         at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]    >         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]    >         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]    >         at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
   [junit4]    >         at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]    >         at 
org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]    >         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]    >         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]    >         at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   [junit4]    >         at 
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   [junit4]    >         at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >   15) Thread[id=20512, name=qtp150412892-20512, state=BLOCKED, 
group=TGRP-FullSolrCloudDistribCmdsTest]
   [junit4]    >         at 
org.apache.log4j.Category.callAppenders(Category.java:204)
   [junit4]    >         at 
org.apache.log4j.Category.forcedLog(Category.java:391)
   [junit4]    >         at org.apache.log4j.Category.log(Category.java:856)
   [junit4]    >         at 
org.slf4j.impl.Log4jLoggerAdapter.warn(Log4jLoggerAdapter.java:400)
   [junit4]    >         at 
org.apache.solr.metrics.SolrMetricManager.closeReporters(SolrMetricManager.java:943)
   [junit4]    >         at 
org.apache.solr.metrics.SolrCoreMetricManager.close(SolrCoreMetricManager.java:146)
   [junit4]    >         at 
org.apache.solr.core.SolrCore.close(SolrCore.java:1581)
   [junit4]    >         at 
org.apache.solr.servlet.HttpSolrCall.destroy(HttpSolrCall.java:579)
   [junit4]    >         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:391)
   [junit4]    >         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
   [junit4]    >         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]    >         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]    >         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]    >         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]    >         at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]    >         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]    >         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]    >         at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]    >         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]    >         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]    >         at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
   [junit4]    >         at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]    >         at 
org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]    >         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]    >         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]    >         at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   [junit4]    >         at 
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   [junit4]    >         at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([312D14671E6B3DDE]:0)
   [junit4] Completed [505/727 (2!)] on J0 in 1699.93s, 1 test, 3 errors <<< 
FAILURES!

[...truncated 675 lines...]
   [junit4] JVM J0: stdout was not empty, see: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/temp/junit4-J0-20170712_045056_5712924593271098990980.sysout
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/heapdumps/java_pid18990.hprof
 ...
   [junit4] Heap dump file created [675978761 bytes in 17.155 secs]
   [junit4] <<< JVM J0: EOF ----

[...truncated 7505 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/build.xml:818:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/build.xml:762:
 Some of the tests produced a heap dump, but did not fail. Maybe a suppressed 
OutOfMemoryError? Dumps created:
* java_pid18990.hprof

Total time: 376 minutes 52 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org