[JENKINS] Lucene-Solr-NightlyTests-7.3 - Build # 14 - Failure

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.3/14/

5 tests failed.
FAILED:  org.apache.lucene.index.TestIndexSorting.testRandom3

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
        at 
__randomizedtesting.SeedInfo.seed([971A6EEBE36729F2:35C22031879500F4]:0)
        at 
org.apache.lucene.codecs.memory.DirectDocValuesProducer.loadNumeric(DirectDocValuesProducer.java:326)
        at 
org.apache.lucene.codecs.memory.DirectDocValuesProducer.loadSortedNumeric(DirectDocValuesProducer.java:511)
        at 
org.apache.lucene.codecs.memory.DirectDocValuesProducer.getSortedNumeric(DirectDocValuesProducer.java:468)
        at 
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.getSortedNumeric(PerFieldDocValuesFormat.java:327)
        at 
org.apache.lucene.index.CodecReader.getSortedNumericDocValues(CodecReader.java:168)
        at 
org.apache.lucene.index.SimpleMergedSegmentWarmer.warm(SimpleMergedSegmentWarmer.java:69)
        at 
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4590)
        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4083)
        at 
org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
        at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2247)
        at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5098)
        at 
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1732)
        at 
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1464)
        at 
org.apache.lucene.index.TestIndexSorting.testRandom3(TestIndexSorting.java:2299)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)


FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Error Message:
27 threads leaked from SUITE scope at 
org.apache.solr.cloud.FullSolrCloudDistribCmdsTest:     1) Thread[id=27573, 
name=searcherExecutor-5830-thread-1, state=WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
        at java.lang.Thread.run(Thread.java:748)    2) Thread[id=27582, 
name=searcherExecutor-5836-thread-1, state=WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
        at java.lang.Thread.run(Thread.java:748)    3) Thread[id=28615, 
name=qtp2010364773-28615, state=RUNNABLE, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.pushFrame(SegmentTermsEnum.java:241)
         at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:471)
         at 
org.apache.lucene.index.AssertingLeafReader$AssertingTermsEnum.seekExact(AssertingLeafReader.java:258)
         at 
org.apache.lucene.search.TermQuery$TermWeight.getTermsEnum(TermQuery.java:127)  
       at 
org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:90)         
at org.apache.lucene.search.Weight.scorerSupplier(Weight.java:113)         at 
org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:329)   
      at org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:295)  
       at org.apache.lucene.search.Weight.bulkScorer(Weight.java:147)         
at org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:289)    
     at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:657)   
      at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)  
       at 
org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:215)
         at 
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1602)
         at 
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1419)
         at 
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:584)     
    at 
org.apache.solr.handler.component.QueryComponent.doProcessUngroupedSearch(QueryComponent.java:1435)
         at 
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:375)
         at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195)
         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)         
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711)         
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:517)         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:384)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:330)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)      
   at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
         at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)       
  at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)   
      at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:455)  
       at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) 
        at org.eclipse.jetty.server.Server.handle(Server.java:530)         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)     
    at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)   
      at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)   
      at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
         at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
         at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) 
        at java.lang.Thread.run(Thread.java:748)    4) Thread[id=27160, 
name=qtp57643525-27160, state=RUNNABLE, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnumFrame.<init>(SegmentTermsEnumFrame.java:47)
         at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.getFrame(SegmentTermsEnum.java:215)
         at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.pushFrame(SegmentTermsEnum.java:241)
         at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:471)
         at 
org.apache.lucene.index.AssertingLeafReader$AssertingTermsEnum.seekExact(AssertingLeafReader.java:258)
         at 
org.apache.lucene.search.TermQuery$TermWeight.getTermsEnum(TermQuery.java:127)  
       at 
org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:90)         
at org.apache.lucene.search.Weight.bulkScorer(Weight.java:147)         at 
org.apache.lucene.search.BooleanWeight.optionalBulkScorer(BooleanWeight.java:157)
         at 
org.apache.lucene.search.BooleanWeight.booleanScorer(BooleanWeight.java:214)    
     at 
org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:283)       
  at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:657)      
   at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)     
    at 
org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:215)
         at 
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1602)
         at 
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1419)
         at 
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:584)     
    at 
org.apache.solr.handler.component.QueryComponent.doProcessUngroupedSearch(QueryComponent.java:1435)
         at 
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:375)
         at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195)
         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)         
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711)         
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:517)         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:384)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:330)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)      
   at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
         at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)       
  at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)   
      at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:455)  
       at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) 
        at org.eclipse.jetty.server.Server.handle(Server.java:530)         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)     
    at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)   
      at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)   
      at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
         at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
         at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) 
        at java.lang.Thread.run(Thread.java:748)    5) Thread[id=27303, 
name=qtp1576252460-27303, state=RUNNABLE, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at 
org.apache.lucene.codecs.lucene50.Lucene50PostingsReader.newTermState(Lucene50PostingsReader.java:145)
         at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnumFrame.<init>(SegmentTermsEnumFrame.java:98)
         at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.getFrame(SegmentTermsEnum.java:215)
         at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.pushFrame(SegmentTermsEnum.java:241)
         at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:471)
         at 
org.apache.lucene.index.AssertingLeafReader$AssertingTermsEnum.seekExact(AssertingLeafReader.java:258)
         at 
org.apache.lucene.search.TermQuery$TermWeight.getTermsEnum(TermQuery.java:127)  
       at 
org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:90)         
at org.apache.lucene.search.Weight.bulkScorer(Weight.java:147)         at 
org.apache.lucene.search.BooleanWeight.optionalBulkScorer(BooleanWeight.java:157)
         at 
org.apache.lucene.search.BooleanWeight.booleanScorer(BooleanWeight.java:214)    
     at 
org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:283)       
  at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:657)      
   at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)     
    at 
org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:215)
         at 
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1602)
         at 
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1419)
         at 
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:584)     
    at 
org.apache.solr.handler.component.QueryComponent.doProcessUngroupedSearch(QueryComponent.java:1435)
         at 
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:375)
         at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195)
         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)         
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711)         
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:517)         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:384)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:330)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)      
   at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
         at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)       
  at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)   
      at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:455)  
       at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) 
        at org.eclipse.jetty.server.Server.handle(Server.java:530)         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)     
    at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)   
      at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)   
      at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
         at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
         at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) 
        at java.lang.Thread.run(Thread.java:748)    6) Thread[id=27600, 
name=searcherExecutor-5868-thread-1, state=WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
        at java.lang.Thread.run(Thread.java:748)    7) Thread[id=27191, 
name=qtp534775150-27191, state=RUNNABLE, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.getFrame(SegmentTermsEnum.java:215)
         at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.pushFrame(SegmentTermsEnum.java:241)
         at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:471)
         at 
org.apache.lucene.index.AssertingLeafReader$AssertingTermsEnum.seekExact(AssertingLeafReader.java:258)
         at 
org.apache.lucene.search.TermQuery$TermWeight.getTermsEnum(TermQuery.java:127)  
       at 
org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:90)         
at org.apache.lucene.search.Weight.scorerSupplier(Weight.java:113)         at 
org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:329)   
      at org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:295)  
       at org.apache.lucene.search.Weight.bulkScorer(Weight.java:147)         
at org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:289)    
     at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:657)   
      at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)  
       at 
org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:215)
         at 
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1602)
         at 
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1419)
         at 
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:584)     
    at 
org.apache.solr.handler.component.QueryComponent.doProcessUngroupedSearch(QueryComponent.java:1435)
         at 
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:375)
         at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195)
         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)         
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711)         
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:517)         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:384)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:330)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)      
   at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
         at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)       
  at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)   
      at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:455)  
       at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) 
        at org.eclipse.jetty.server.Server.handle(Server.java:530)         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)     
    at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)   
      at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)   
      at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
         at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
         at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) 
        at java.lang.Thread.run(Thread.java:748)    8) Thread[id=27548, 
name=qtp1576252460-27548, state=RUNNABLE, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnumFrame.<init>(SegmentTermsEnumFrame.java:54)
         at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.getFrame(SegmentTermsEnum.java:215)
         at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.pushFrame(SegmentTermsEnum.java:241)
         at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:471)
         at 
org.apache.lucene.index.AssertingLeafReader$AssertingTermsEnum.seekExact(AssertingLeafReader.java:258)
         at 
org.apache.lucene.search.TermQuery$TermWeight.getTermsEnum(TermQuery.java:127)  
       at 
org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:90)         
at org.apache.lucene.search.Weight.bulkScorer(Weight.java:147)         at 
org.apache.lucene.search.BooleanWeight.optionalBulkScorer(BooleanWeight.java:157)
         at 
org.apache.lucene.search.BooleanWeight.booleanScorer(BooleanWeight.java:214)    
     at 
org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:283)       
  at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:657)      
   at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)     
    at 
org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:215)
         at 
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1602)
         at 
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1419)
         at 
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:584)     
    at 
org.apache.solr.handler.component.QueryComponent.doProcessUngroupedSearch(QueryComponent.java:1435)
         at 
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:375)
         at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195)
         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)         
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711)         
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:517)         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:384)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:330)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)      
   at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
         at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)       
  at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)   
      at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:455)  
       at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) 
        at org.eclipse.jetty.server.Server.handle(Server.java:530)         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)     
    at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)   
      at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)   
      at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
         at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
         at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) 
        at java.lang.Thread.run(Thread.java:748)    9) Thread[id=27571, 
name=searcherExecutor-5834-thread-1, state=WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
        at java.lang.Thread.run(Thread.java:748)   10) Thread[id=27602, 
name=searcherExecutor-5869-thread-1, state=WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
        at java.lang.Thread.run(Thread.java:748)   11) Thread[id=27301, 
name=qtp1576252460-27301, state=RUNNABLE, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at 
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.terms(PerFieldPostingsFormat.java:316)
         at org.apache.lucene.index.CodecReader.terms(CodecReader.java:106)     
    at 
org.apache.lucene.index.FilterLeafReader.terms(FilterLeafReader.java:352)       
  at 
org.apache.lucene.index.ExitableDirectoryReader$ExitableFilterAtomicReader.terms(ExitableDirectoryReader.java:82)
         at 
org.apache.lucene.search.TermQuery$TermWeight.getTermsEnum(TermQuery.java:122)  
       at 
org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:90)         
at org.apache.lucene.search.Weight.bulkScorer(Weight.java:147)         at 
org.apache.lucene.search.BooleanWeight.optionalBulkScorer(BooleanWeight.java:157)
         at 
org.apache.lucene.search.BooleanWeight.booleanScorer(BooleanWeight.java:214)    
     at 
org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:283)       
  at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:657)      
   at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)     
    at 
org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:215)
         at 
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1602)
         at 
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1419)
         at 
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:584)     
    at 
org.apache.solr.handler.component.QueryComponent.doProcessUngroupedSearch(QueryComponent.java:1435)
         at 
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:375)
         at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195)
         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)         
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711)         
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:517)         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:384)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:330)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)      
   at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
         at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)       
  at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)   
      at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:455)  
       at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) 
        at org.eclipse.jetty.server.Server.handle(Server.java:530)         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)     
    at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)   
      at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)   
      at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
         at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
         at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) 
        at java.lang.Thread.run(Thread.java:748)   12) Thread[id=28513, 
name=qtp1576252460-28513, state=RUNNABLE, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at 
org.apache.lucene.codecs.lucene50.Lucene50PostingsReader.newTermState(Lucene50PostingsReader.java:145)
         at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnumFrame.<init>(SegmentTermsEnumFrame.java:98)
         at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.getFrame(SegmentTermsEnum.java:215)
         at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.pushFrame(SegmentTermsEnum.java:241)
         at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:471)
         at 
org.apache.lucene.index.AssertingLeafReader$AssertingTermsEnum.seekExact(AssertingLeafReader.java:258)
         at 
org.apache.lucene.search.TermQuery$TermWeight.getTermsEnum(TermQuery.java:127)  
       at 
org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:90)         
at org.apache.lucene.search.Weight.bulkScorer(Weight.java:147)         at 
org.apache.lucene.search.BooleanWeight.optionalBulkScorer(BooleanWeight.java:157)
         at 
org.apache.lucene.search.BooleanWeight.booleanScorer(BooleanWeight.java:214)    
     at 
org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:283)       
  at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:657)      
   at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)     
    at 
org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:215)
         at 
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1602)
         at 
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1419)
         at 
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:584)     
    at 
org.apache.solr.handler.component.QueryComponent.doProcessUngroupedSearch(QueryComponent.java:1435)
         at 
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:375)
         at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195)
         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)         
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711)         
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:517)         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:384)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:330)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)      
   at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
         at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)       
  at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)   
      at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:455)  
       at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) 
        at org.eclipse.jetty.server.Server.handle(Server.java:530)         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)     
    at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)   
      at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)   
      at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
         at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
         at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) 
        at java.lang.Thread.run(Thread.java:748)   13) Thread[id=27101, 
name=qtp2010364773-27101, state=RUNNABLE, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at 
java.lang.Object.hashCode(Native Method)         at 
java.util.WeakHashMap.hash(WeakHashMap.java:298)         at 
java.util.WeakHashMap.get(WeakHashMap.java:396)         at 
org.apache.solr.uninverting.FieldCacheImpl$Cache.get(FieldCacheImpl.java:180)   
      at 
org.apache.solr.uninverting.FieldCacheImpl.getTermsIndex(FieldCacheImpl.java:899)
         at 
org.apache.solr.uninverting.FieldCacheImpl.getTermsIndex(FieldCacheImpl.java:879)
         at 
org.apache.solr.uninverting.UninvertingReader.getSortedDocValues(UninvertingReader.java:347)
         at 
org.apache.lucene.index.FilterLeafReader.getSortedDocValues(FilterLeafReader.java:378)
         at org.apache.lucene.index.DocValues.getSorted(DocValues.java:390)     
    at 
org.apache.lucene.search.FieldComparator$TermOrdValComparator.getSortedDocValues(FieldComparator.java:709)
         at 
org.apache.lucene.search.FieldComparator$TermOrdValComparator.getLeafComparator(FieldComparator.java:714)
         at 
org.apache.lucene.search.FieldValueHitQueue.getComparators(FieldValueHitQueue.java:180)
         at 
org.apache.lucene.search.TopFieldCollector$SimpleFieldCollector.getLeafCollector(TopFieldCollector.java:122)
         at 
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:651)         
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)        
 at 
org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:215)
         at 
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1602)
         at 
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1419)
         at 
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:584)     
    at 
org.apache.solr.handler.component.QueryComponent.doProcessUngroupedSearch(QueryComponent.java:1435)
         at 
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:375)
         at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195)
         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)         
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711)         
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:517)         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:384)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:330)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)      
   at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
         at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)       
  at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)   
      at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:455)  
       at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) 
        at org.eclipse.jetty.server.Server.handle(Server.java:530)         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)     
    at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)   
      at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)   
      at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
         at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
         at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) 
        at java.lang.Thread.run(Thread.java:748)   14) Thread[id=27528, 
name=qtp534775150-27528, state=RUNNABLE, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnumFrame.<init>(SegmentTermsEnumFrame.java:90)
         at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.getFrame(SegmentTermsEnum.java:215)
         at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.pushFrame(SegmentTermsEnum.java:241)
         at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:471)
         at 
org.apache.lucene.index.AssertingLeafReader$AssertingTermsEnum.seekExact(AssertingLeafReader.java:258)
         at 
org.apache.lucene.search.TermQuery$TermWeight.getTermsEnum(TermQuery.java:127)  
       at 
org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:90)         
at org.apache.lucene.search.Weight.bulkScorer(Weight.java:147)         at 
org.apache.lucene.search.BooleanWeight.optionalBulkScorer(BooleanWeight.java:157)
         at 
org.apache.lucene.search.BooleanWeight.booleanScorer(BooleanWeight.java:214)    
     at 
org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:283)       
  at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:657)      
   at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)     
    at 
org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:215)
         at 
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1602)
         at 
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1419)
         at 
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:584)     
    at 
org.apache.solr.handler.component.QueryComponent.doProcessUngroupedSearch(QueryComponent.java:1435)
         at 
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:375)
         at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195)
         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)         
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711)         
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:517)         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:384)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:330)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)      
   at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
         at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)       
  at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)   
      at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:455)  
       at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) 
        at org.eclipse.jetty.server.Server.handle(Server.java:530)         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)     
    at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)   
      at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)   
      at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
         at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
         at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) 
        at java.lang.Thread.run(Thread.java:748)   15) Thread[id=28637, 
name=qtp2010364773-28637, state=RUNNABLE, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at 
org.apache.lucene.codecs.blocktree.FieldReader.iterator(FieldReader.java:156)   
      at 
org.apache.lucene.index.FilterLeafReader$FilterTerms.iterator(FilterLeafReader.java:107)
         at 
org.apache.lucene.index.AssertingLeafReader$AssertingTerms.iterator(AssertingLeafReader.java:139)
         at 
org.apache.lucene.search.TermQuery$TermWeight.getTermsEnum(TermQuery.java:126)  
       at 
org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:90)         
at org.apache.lucene.search.Weight.bulkScorer(Weight.java:147)         at 
org.apache.lucene.search.BooleanWeight.optionalBulkScorer(BooleanWeight.java:157)
         at 
org.apache.lucene.search.BooleanWeight.booleanScorer(BooleanWeight.java:214)    
     at 
org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:283)       
  at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:657)      
   at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)     
    at 
org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:215)
         at 
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1602)
         at 
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1419)
         at 
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:584)     
    at 
org.apache.solr.handler.component.QueryComponent.doProcessUngroupedSearch(QueryComponent.java:1435)
         at 
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:375)
         at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195)
         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)         
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711)         
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:517)         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:384)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:330)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)      
   at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
         at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)       
  at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)   
      at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:455)  
       at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) 
        at org.eclipse.jetty.server.Server.handle(Server.java:530)         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)     
    at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)   
      at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)   
      at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
         at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
         at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) 
        at java.lang.Thread.run(Thread.java:748)   16) Thread[id=27297, 
name=qtp1576252460-27297, state=RUNNABLE, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.getFrame(SegmentTermsEnum.java:215)
         at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.pushFrame(SegmentTermsEnum.java:241)
         at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:471)
         at 
org.apache.lucene.index.AssertingLeafReader$AssertingTermsEnum.seekExact(AssertingLeafReader.java:258)
         at 
org.apache.lucene.search.TermQuery$TermWeight.getTermsEnum(TermQuery.java:127)  
       at 
org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:90)         
at org.apache.lucene.search.Weight.bulkScorer(Weight.java:147)         at 
org.apache.lucene.search.BooleanWeight.optionalBulkScorer(BooleanWeight.java:157)
         at 
org.apache.lucene.search.BooleanWeight.booleanScorer(BooleanWeight.java:214)    
     at 
org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:283)       
  at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:657)      
   at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)     
    at 
org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:215)
         at 
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1602)
         at 
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1419)
         at 
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:584)     
    at 
org.apache.solr.handler.component.QueryComponent.doProcessUngroupedSearch(QueryComponent.java:1435)
         at 
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:375)
         at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195)
         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)         
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711)         
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:517)         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:384)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:330)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)      
   at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
         at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)       
  at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)   
      at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:455)  
       at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) 
        at org.eclipse.jetty.server.Server.handle(Server.java:530)         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)     
    at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)   
      at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)   
      at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
         at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
         at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) 
        at java.lang.Thread.run(Thread.java:748)   17) Thread[id=27599, 
name=searcherExecutor-5872-thread-1, state=WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
        at java.lang.Thread.run(Thread.java:748)   18) Thread[id=27579, 
name=searcherExecutor-5832-thread-1, state=WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
        at java.lang.Thread.run(Thread.java:748)   19) Thread[id=27605, 
name=searcherExecutor-5904-thread-1, state=WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
        at java.lang.Thread.run(Thread.java:748)   20) Thread[id=27158, 
name=qtp57643525-27158, state=RUNNABLE, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnumFrame.<init>(SegmentTermsEnumFrame.java:48)
         at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.<init>(SegmentTermsEnum.java:77)
         at 
org.apache.lucene.codecs.blocktree.FieldReader.iterator(FieldReader.java:156)   
      at 
org.apache.lucene.index.FilterLeafReader$FilterTerms.iterator(FilterLeafReader.java:107)
         at 
org.apache.lucene.index.AssertingLeafReader$AssertingTerms.iterator(AssertingLeafReader.java:139)
         at 
org.apache.lucene.search.TermQuery$TermWeight.getTermsEnum(TermQuery.java:126)  
       at 
org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:90)         
at org.apache.lucene.search.Weight.scorerSupplier(Weight.java:113)         at 
org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:329)   
      at org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:295)  
       at org.apache.lucene.search.Weight.bulkScorer(Weight.java:147)         
at org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:289)    
     at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:657)   
      at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)  
       at 
org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:215)
         at 
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1602)
         at 
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1419)
         at 
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:584)     
    at 
org.apache.solr.handler.component.QueryComponent.doProcessUngroupedSearch(QueryComponent.java:1435)
         at 
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:375)
         at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195)
         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)         
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711)         
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:517)         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:384)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:330)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)      
   at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
         at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)       
  at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)   
      at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:455)  
       at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) 
        at org.eclipse.jetty.server.Server.handle(Server.java:530)         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)     
    at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)   
      at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)   
      at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
         at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
         at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) 
        at java.lang.Thread.run(Thread.java:748)   21) Thread[id=27580, 
name=searcherExecutor-5833-thread-1, state=WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
        at java.lang.Thread.run(Thread.java:748)   22) Thread[id=27603, 
name=searcherExecutor-5870-thread-1, state=WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
        at java.lang.Thread.run(Thread.java:748)   23) Thread[id=27264, 
name=qtp1130521860-27264, state=RUNNABLE, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnumFrame.<init>(SegmentTermsEnumFrame.java:47)
         at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.getFrame(SegmentTermsEnum.java:215)
         at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.pushFrame(SegmentTermsEnum.java:241)
         at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:471)
         at 
org.apache.lucene.index.AssertingLeafReader$AssertingTermsEnum.seekExact(AssertingLeafReader.java:258)
         at 
org.apache.lucene.search.TermQuery$TermWeight.getTermsEnum(TermQuery.java:127)  
       at 
org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:90)         
at org.apache.lucene.search.Weight.bulkScorer(Weight.java:147)         at 
org.apache.lucene.search.BooleanWeight.optionalBulkScorer(BooleanWeight.java:157)
         at 
org.apache.lucene.search.BooleanWeight.booleanScorer(BooleanWeight.java:214)    
     at 
org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:283)       
  at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:657)      
   at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)     
    at 
org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:215)
         at 
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1602)
         at 
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1419)
         at 
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:584)     
    at 
org.apache.solr.handler.component.QueryComponent.doProcessUngroupedSearch(QueryComponent.java:1435)
         at 
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:375)
         at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195)
         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)         
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711)         
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:517)         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:384)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:330)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)      
   at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
         at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)       
  at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)   
      at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:455)  
       at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) 
        at org.eclipse.jetty.server.Server.handle(Server.java:530)         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)     
    at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)   
      at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)   
      at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
         at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
         at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) 
        at java.lang.Thread.run(Thread.java:748)   24) Thread[id=27587, 
name=searcherExecutor-5847-thread-1, state=WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
        at java.lang.Thread.run(Thread.java:748)   25) Thread[id=27105, 
name=qtp2010364773-27105, state=RUNNABLE, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at 
java.util.Arrays.copyOf(Arrays.java:3332)         at 
java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
         at 
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)         
at java.lang.StringBuilder.append(StringBuilder.java:136)         at 
org.apache.lucene.store.MockIndexInputWrapper.<init>(MockIndexInputWrapper.java:40)
         at 
org.apache.lucene.store.MockIndexInputWrapper.clone(MockIndexInputWrapper.java:89)
         at 
org.apache.lucene.store.MockIndexInputWrapper.clone(MockIndexInputWrapper.java:29)
         at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.initIndexInput(SegmentTermsEnum.java:115)
         at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnumFrame.loadBlock(SegmentTermsEnumFrame.java:145)
         at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:507)
         at 
org.apache.lucene.index.AssertingLeafReader$AssertingTermsEnum.seekExact(AssertingLeafReader.java:258)
         at 
org.apache.lucene.search.TermQuery$TermWeight.getTermsEnum(TermQuery.java:127)  
       at 
org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:90)         
at org.apache.lucene.search.Weight.scorerSupplier(Weight.java:113)         at 
org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:329)   
      at org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:295)  
       at org.apache.lucene.search.Weight.bulkScorer(Weight.java:147)         
at org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:289)    
     at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:657)   
      at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)  
       at 
org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:215)
         at 
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1602)
         at 
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1419)
         at 
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:584)     
    at 
org.apache.solr.handler.component.QueryComponent.doProcessUngroupedSearch(QueryComponent.java:1435)
         at 
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:375)
         at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195)
         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)         
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711)         
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:517)         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:384)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:330)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)      
   at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
         at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)       
  at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)   
      at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:455)  
       at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) 
        at org.eclipse.jetty.server.Server.handle(Server.java:530)         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)     
    at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)   
      at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)   
      at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
         at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
         at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) 
        at java.lang.Thread.run(Thread.java:748)   26) Thread[id=27604, 
name=searcherExecutor-5863-thread-1, state=WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
        at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 27 threads leaked from 
SUITE scope at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest: 
   1) Thread[id=27573, name=searcherExecutor-5830-thread-1, state=WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=27582, name=searcherExecutor-5836-thread-1, state=WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=28615, name=qtp2010364773-28615, state=RUNNABLE, 
group=TGRP-FullSolrCloudDistribCmdsTest]
        at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.pushFrame(SegmentTermsEnum.java:241)
        at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:471)
        at 
org.apache.lucene.index.AssertingLeafReader$AssertingTermsEnum.seekExact(AssertingLeafReader.java:258)
        at 
org.apache.lucene.search.TermQuery$TermWeight.getTermsEnum(TermQuery.java:127)
        at 
org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:90)
        at org.apache.lucene.search.Weight.scorerSupplier(Weight.java:113)
        at 
org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:329)
        at org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:295)
        at org.apache.lucene.search.Weight.bulkScorer(Weight.java:147)
        at 
org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:289)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:657)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)
        at 
org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:215)
        at 
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1602)
        at 
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1419)
        at 
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:584)
        at 
org.apache.solr.handler.component.QueryComponent.doProcessUngroupedSearch(QueryComponent.java:1435)
        at 
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:375)
        at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
        at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:517)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:384)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:330)
        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
        at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
        at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
        at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
        at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
        at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
        at org.eclipse.jetty.server.handler.Cont

[...truncated too long message...]

 >         at 
 > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >   24) Thread[id=27587, name=searcherExecutor-5847-thread-1, 
state=WAITING, group=TGRP-FullSolrCloudDistribCmdsTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >   25) Thread[id=27105, name=qtp2010364773-27105, 
state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
   [junit4]    >         at 
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.<init>(SegmentTermsEnum.java:88)
   [junit4]    >         at 
org.apache.lucene.codecs.blocktree.FieldReader.iterator(FieldReader.java:156)
   [junit4]    >         at 
org.apache.lucene.index.FilterLeafReader$FilterTerms.iterator(FilterLeafReader.java:107)
   [junit4]    >         at 
org.apache.lucene.index.AssertingLeafReader$AssertingTerms.iterator(AssertingLeafReader.java:139)
   [junit4]    >         at 
org.apache.lucene.search.TermQuery$TermWeight.getTermsEnum(TermQuery.java:126)
   [junit4]    >         at 
org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:90)
   [junit4]    >         at 
org.apache.lucene.search.Weight.scorerSupplier(Weight.java:113)
   [junit4]    >         at 
org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:329)
   [junit4]    >         at 
org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:295)
   [junit4]    >         at 
org.apache.lucene.search.Weight.bulkScorer(Weight.java:147)
   [junit4]    >         at 
org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:289)
   [junit4]    >         at 
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:657)
   [junit4]    >         at 
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)
   [junit4]    >         at 
org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:215)
   [junit4]    >         at 
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1602)
   [junit4]    >         at 
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1419)
   [junit4]    >         at 
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:584)
   [junit4]    >         at 
org.apache.solr.handler.component.QueryComponent.doProcessUngroupedSearch(QueryComponent.java:1435)
   [junit4]    >         at 
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:375)
   [junit4]    >         at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
   [junit4]    >         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195)
   [junit4]    >         at 
org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)
   [junit4]    >         at 
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711)
   [junit4]    >         at 
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:517)
   [junit4]    >         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:384)
   [junit4]    >         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:330)
   [junit4]    >         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
   [junit4]    >         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]    >         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
   [junit4]    >         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
   [junit4]    >         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
   [junit4]    >         at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
   [junit4]    >         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
   [junit4]    >         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
   [junit4]    >         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
   [junit4]    >         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
   [junit4]    >         at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
   [junit4]    >         at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
   [junit4]    >         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
   [junit4]    >         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]    >         at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:455)
   [junit4]    >         at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
   [junit4]    >         at 
org.eclipse.jetty.server.Server.handle(Server.java:530)
   [junit4]    >         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)
   [junit4]    >         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)
   [junit4]    >         at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
   [junit4]    >         at 
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
   [junit4]    >         at 
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >   26) Thread[id=27604, name=searcherExecutor-5863-thread-1, 
state=WAITING, group=TGRP-FullSolrCloudDistribCmdsTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([EC783F455A1C339]:0)
   [junit4] Completed [550/783 (2!)] on J0 in 4093.77s, 1 test, 3 errors <<< 
FAILURES!

[...truncated 725 lines...]
   [junit4] JVM J0: stdout was not empty, see: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.3/checkout/solr/build/solr-core/test/temp/junit4-J0-20180330_121655_4155397653190135330835.sysout
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.3/checkout/heapdumps/java_pid28550.hprof
 ...
   [junit4] Heap dump file created [698161146 bytes in 17.672 secs]
   [junit4] <<< JVM J0: EOF ----

[...truncated 9294 lines...]
BUILD FAILED
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.3/checkout/build.xml:636:
 The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.3/checkout/build.xml:570:
 Some of the tests produced a heap dump, but did not fail. Maybe a suppressed 
OutOfMemoryError? Dumps created:
* java_pid28550.hprof
* java_pid936.hprof

Total time: 485 minutes 11 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org