Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-main/587/

111 tests failed.
FAILED:  org.apache.solr.ltr.TestLTROnSolrCloud.testSimpleQuery

Error Message:
org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException: Error 
from server at https://127.0.0.1:42335/solr: Underlying core creation failed 
while creating collection: collection1

Stack Trace:
org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException: Error 
from server at https://127.0.0.1:42335/solr: Underlying core creation failed 
while creating collection: collection1
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:698)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:269)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:251)
        at 
org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:390)
        at 
org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:360)
        at 
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1164)
        at 
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:927)
        at 
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:861)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:229)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:246)
        at 
org.apache.solr.ltr.TestLTROnSolrCloud.createCollection(TestLTROnSolrCloud.java:226)
        at 
org.apache.solr.ltr.TestLTROnSolrCloud.setupSolrCluster(TestLTROnSolrCloud.java:208)
        at 
org.apache.solr.ltr.TestLTROnSolrCloud.setUp(TestLTROnSolrCloud.java:59)
        at jdk.internal.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
        at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:976)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:470)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:826)
        at java.base/java.lang.Thread.run(Thread.java:834)


FAILED:  org.apache.solr.ltr.TestLTROnSolrCloud.classMethod

Error Message:
com.carrotsearch.randomizedtesting.ThreadLeakError: 65 threads leaked from 
SUITE scope at org.apache.solr.ltr.TestLTROnSolrCloud: 
   1) Thread[id=500, name=Session-HouseKeeper-3833faaa-1, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
   2) Thread[id=526, name=qtp1052695412-526, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:842)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:892)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
   3) Thread[id=469, name=NIOWorkerThread-1, state=WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at 
[email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
   4) Thread[id=576, name=coreContainerWorkExecutor-320-thread-1, 
state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
   5) Thread[id=467, name=ProcessThread(sid:0 cport:38981):, state=WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at 
[email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at 
app//org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:140)
   6) Thread[id=511, 
name=qtp89046542-511-acceptor-0@2c3f0108-ServerConnector@3afa237a{ssl, (ssl, 
alpn, http/1.1, h2)}{127.0.0.1:43559}, state=RUNNABLE, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/sun.nio.ch.ServerSocketChannelImpl.accept0(Native 
Method)
        at 
[email protected]/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
        at 
[email protected]/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
        at 
app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:388)
        at 
app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:702)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
   7) Thread[id=523, 
name=qtp1052695412-523-acceptor-0@4b405e1a-ServerConnector@39e928cf{ssl, (ssl, 
alpn, http/1.1, h2)}{127.0.0.1:42897}, state=RUNNABLE, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/sun.nio.ch.ServerSocketChannelImpl.accept0(Native 
Method)
        at 
[email protected]/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
        at 
[email protected]/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
        at 
app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:388)
        at 
app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:702)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
   8) Thread[id=493, name=qtp1141594396-493, state=RUNNABLE, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/sun.nio.ch.EPoll.wait(Native Method)
        at 
[email protected]/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at 
[email protected]/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at 
[email protected]/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at 
app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:149)
        at 
app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:156)
        at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:572)
        at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:509)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
        at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$269/0x0000000100441c40.run(Unknown
 Source)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
   9) Thread[id=515, name=qtp89046542-515, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:842)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:892)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  10) Thread[id=481, name=NIOWorkerThread-8, state=WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at 
[email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  11) Thread[id=494, 
name=qtp1251541303-494-acceptor-0@5f8b8b87-ServerConnector@7f1c50d{ssl, (ssl, 
alpn, http/1.1, h2)}{127.0.0.1:37507}, state=RUNNABLE, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/sun.nio.ch.ServerSocketChannelImpl.accept0(Native 
Method)
        at 
[email protected]/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
        at 
[email protected]/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
        at 
app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:388)
        at 
app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:702)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  12) Thread[id=496, name=qtp1251541303-496, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:842)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:892)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  13) Thread[id=508, name=qtp1141594396-508, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:842)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:892)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  14) Thread[id=517, name=Session-HouseKeeper-6c2ffbc2-1, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  15) Thread[id=578, name=coreContainerWorkExecutor-350-thread-1, 
state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  16) Thread[id=525, name=qtp1052695412-525, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:842)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:892)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  17) Thread[id=503, name=qtp1141594396-503, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:842)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:892)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  18) Thread[id=597, name=Connector-Scheduler-3afa237a-1, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  19) Thread[id=527, name=qtp1052695412-527, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:842)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:892)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  20) Thread[id=477, name=NIOWorkerThread-6, state=WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at 
[email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  21) Thread[id=529, name=Session-HouseKeeper-4481e712-1, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  22) Thread[id=579, name=CloserThread, state=WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/java.lang.Object.wait(Native Method)
        at [email protected]/java.lang.Object.wait(Object.java:328)
        at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2326)
  23) Thread[id=600, name=Connector-Scheduler-39e928cf-1, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  24) Thread[id=507, name=qtp1052695412-507, state=RUNNABLE, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/sun.nio.ch.EPoll.wait(Native Method)
        at 
[email protected]/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at 
[email protected]/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at 
[email protected]/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at 
app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:149)
        at 
app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:156)
        at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:572)
        at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:509)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
        at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$269/0x0000000100441c40.run(Unknown
 Source)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  25) Thread[id=573, name=CloserThread, state=WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/java.lang.Object.wait(Native Method)
        at [email protected]/java.lang.Object.wait(Object.java:328)
        at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2326)
  26) Thread[id=524, name=qtp1052695412-524, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:842)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:892)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  27) Thread[id=505, name=qtp1141594396-505, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:842)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:892)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  28) Thread[id=514, name=qtp89046542-514, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:842)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:892)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  29) Thread[id=522, name=qtp1052695412-522, state=RUNNABLE, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/sun.nio.ch.EPoll.wait(Native Method)
        at 
[email protected]/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at 
[email protected]/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at 
[email protected]/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at 
app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:149)
        at 
app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:156)
        at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:572)
        at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:509)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
        at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$269/0x0000000100441c40.run(Unknown
 Source)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  30) Thread[id=465, name=SessionTracker, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/java.lang.Thread.sleep(Native Method)
        at 
app//org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:157)
  31) Thread[id=501, name=qtp1141594396-501, state=RUNNABLE, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/sun.nio.ch.EPoll.wait(Native Method)
        at 
[email protected]/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at 
[email protected]/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at 
[email protected]/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at 
app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:149)
        at 
app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:156)
        at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:572)
        at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:509)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
        at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$269/0x0000000100441c40.run(Unknown
 Source)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  32) Thread[id=461, name=ZkTestServer Run Thread, state=WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/java.lang.Object.wait(Native Method)
        at [email protected]/java.lang.Thread.join(Thread.java:1305)
        at [email protected]/java.lang.Thread.join(Thread.java:1379)
        at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:929)
        at 
app//org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:348)
        at app//org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:579)
  33) Thread[id=506, name=qtp1141594396-506, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:842)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:892)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  34) Thread[id=468, name=RequestThrottler, state=WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at 
[email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at 
app//org.apache.zookeeper.server.RequestThrottler.run(RequestThrottler.java:139)
  35) Thread[id=463, name=NIOServerCxnFactory.SelectorThread-0, state=RUNNABLE, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/sun.nio.ch.EPoll.wait(Native Method)
        at 
[email protected]/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at 
[email protected]/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at 
[email protected]/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.select(NIOServerCnxnFactory.java:417)
        at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.run(NIOServerCnxnFactory.java:382)
  36) Thread[id=598, name=Connector-Scheduler-7f1c50d-1, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  37) Thread[id=572, name=Core Container Async Task-331-thread-1, 
state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  38) Thread[id=502, 
name=qtp1141594396-502-acceptor-0@7d94a8e6-ServerConnector@152cce0d{ssl, (ssl, 
alpn, http/1.1, h2)}{127.0.0.1:42335}, state=RUNNABLE, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/sun.nio.ch.ServerSocketChannelImpl.accept0(Native 
Method)
        at 
[email protected]/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
        at 
[email protected]/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
        at 
app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:388)
        at 
app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:702)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  39) Thread[id=472, name=NIOWorkerThread-2, state=WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at 
[email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  40) Thread[id=462, name=ConnnectionExpirer, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/java.lang.Thread.sleep(Native Method)
        at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory$ConnectionExpirerThread.run(NIOServerCnxnFactory.java:568)
  41) Thread[id=478, name=NIOWorkerThread-7, state=WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at 
[email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  42) Thread[id=509, name=Session-HouseKeeper-324b07ef-1, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  43) Thread[id=490, name=qtp1251541303-490, state=RUNNABLE, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/sun.nio.ch.EPoll.wait(Native Method)
        at 
[email protected]/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at 
[email protected]/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at 
[email protected]/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at 
app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:149)
        at 
app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:156)
        at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:572)
        at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:509)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
        at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$269/0x0000000100441c40.run(Unknown
 Source)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  44) Thread[id=495, name=qtp1251541303-495, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:842)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:892)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  45) Thread[id=580, name=coreContainerWorkExecutor-340-thread-1, 
state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  46) Thread[id=491, name=qtp1251541303-491, state=RUNNABLE, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/sun.nio.ch.EPoll.wait(Native Method)
        at 
[email protected]/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at 
[email protected]/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at 
[email protected]/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at 
app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:149)
        at 
app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:156)
        at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:572)
        at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:509)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
        at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$269/0x0000000100441c40.run(Unknown
 Source)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  47) Thread[id=575, name=CloserThread, state=WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/java.lang.Object.wait(Native Method)
        at [email protected]/java.lang.Object.wait(Object.java:328)
        at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2326)
  48) Thread[id=492, name=qtp89046542-492, state=RUNNABLE, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/sun.nio.ch.EPoll.wait(Native Method)
        at 
[email protected]/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at 
[email protected]/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at 
[email protected]/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at 
app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:149)
        at 
app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:156)
        at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:572)
        at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:509)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
        at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$269/0x0000000100441c40.run(Unknown
 Source)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  49) Thread[id=528, name=qtp1052695412-528, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:842)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:892)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  50) Thread[id=475, name=NIOWorkerThread-4, state=WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at 
[email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  51) Thread[id=499, name=qtp1251541303-499, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:842)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:892)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  52) Thread[id=466, name=SyncThread:0, state=WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at 
[email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at 
app//org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:170)
  53) Thread[id=588, name=Connector-Scheduler-152cce0d-1, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  54) Thread[id=476, name=NIOWorkerThread-5, state=WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at 
[email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  55) Thread[id=464, name=NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:0, 
state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]
        at [email protected]/sun.nio.ch.EPoll.wait(Native Method)
        at 
[email protected]/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at 
[email protected]/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at 
[email protected]/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread.select(NIOServerCnxnFactory.java:219)
        at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread.run(NIOServerCnxnFactory.java:195)
  56) Thread[id=504, name=qtp1141594396-504, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:842)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:892)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  57) Thread[id=513, name=qtp89046542-513, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:842)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:892)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  58) Thread[id=512, name=qtp89046542-512, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:842)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:892)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  59) Thread[id=473, name=NIOWorkerThread-3, state=WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at 
[email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  60) Thread[id=577, name=CloserThread, state=WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/java.lang.Object.wait(Native Method)
        at [email protected]/java.lang.Object.wait(Object.java:328)
        at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2326)
  61) Thread[id=510, name=qtp89046542-510, state=RUNNABLE, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/sun.nio.ch.EPoll.wait(Native Method)
        at 
[email protected]/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at 
[email protected]/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at 
[email protected]/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at 
app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:149)
        at 
app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:156)
        at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:572)
        at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:509)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
        at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$269/0x0000000100441c40.run(Unknown
 Source)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  62) Thread[id=516, name=qtp89046542-516, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:842)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:892)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  63) Thread[id=497, name=qtp1251541303-497, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:842)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:892)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  64) Thread[id=498, name=qtp1251541303-498, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:842)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:892)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  65) Thread[id=574, name=coreContainerWorkExecutor-330-thread-1, 
state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 65 threads leaked from 
SUITE scope at org.apache.solr.ltr.TestLTROnSolrCloud: 
   1) Thread[id=500, name=Session-HouseKeeper-3833faaa-1, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
   2) Thread[id=526, name=qtp1052695412-526, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:842)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:892)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
   3) Thread[id=469, name=NIOWorkerThread-1, state=WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at 
[email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
   4) Thread[id=576, name=coreContainerWorkExecutor-320-thread-1, 
state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
   5) Thread[id=467, name=ProcessThread(sid:0 cport:38981):, state=WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at 
[email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at 
app//org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:140)
   6) Thread[id=511, 
name=qtp89046542-511-acceptor-0@2c3f0108-ServerConnector@3afa237a{ssl, (ssl, 
alpn, http/1.1, h2)}{127.0.0.1:43559}, state=RUNNABLE, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/sun.nio.ch.ServerSocketChannelImpl.accept0(Native 
Method)
        at 
[email protected]/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
        at 
[email protected]/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
        at 
app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:388)
        at 
app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:702)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
   7) Thread[id=523, 
name=qtp1052695412-523-acceptor-0@4b405e1a-ServerConnector@39e928cf{ssl, (ssl, 
alpn, http/1.1, h2)}{127.0.0.1:42897}, state=RUNNABLE, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/sun.nio.ch.ServerSocketChannelImpl.accept0(Native 
Method)
        at 
[email protected]/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
        at 
[email protected]/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
        at 
app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:388)
        at 
app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:702)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
   8) Thread[id=493, name=qtp1141594396-493, state=RUNNABLE, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/sun.nio.ch.EPoll.wait(Native Method)
        at 
[email protected]/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at 
[email protected]/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at 
[email protected]/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at 
app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:149)
        at 
app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:156)
        at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:572)
        at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:509)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
        at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$269/0x0000000100441c40.run(Unknown
 Source)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
   9) Thread[id=515, name=qtp89046542-515, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:842)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:892)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  10) Thread[id=481, name=NIOWorkerThread-8, state=WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at 
[email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  11) Thread[id=494, 
name=qtp1251541303-494-acceptor-0@5f8b8b87-ServerConnector@7f1c50d{ssl, (ssl, 
alpn, http/1.1, h2)}{127.0.0.1:37507}, state=RUNNABLE, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/sun.nio.ch.ServerSocketChannelImpl.accept0(Native 
Method)
        at 
[email protected]/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
        at 
[email protected]/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
        at 
app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:388)
        at 
app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:702)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  12) Thread[id=496, name=qtp1251541303-496, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:842)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:892)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  13) Thread[id=508, name=qtp1141594396-508, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:842)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:892)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  14) Thread[id=517, name=Session-HouseKeeper-6c2ffbc2-1, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  15) Thread[id=578, name=coreContainerWorkExecutor-350-thread-1, 
state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  16) Thread[id=525, name=qtp1052695412-525, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:842)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:892)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  17) Thread[id=503, name=qtp1141594396-503, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:842)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:892)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  18) Thread[id=597, name=Connector-Scheduler-3afa237a-1, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  19) Thread[id=527, name=qtp1052695412-527, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:842)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:892)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  20) Thread[id=477, name=NIOWorkerThread-6, state=WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at 
[email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  21) Thread[id=529, name=Session-HouseKeeper-4481e712-1, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  22) Thread[id=579, name=CloserThread, state=WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/java.lang.Object.wait(Native Method)
        at [email protected]/java.lang.Object.wait(Object.java:328)
        at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2326)
  23) Thread[id=600, name=Connector-Scheduler-39e928cf-1, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  24) Thread[id=507, name=qtp1052695412-507, state=RUNNABLE, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/sun.nio.ch.EPoll.wait(Native Method)
        at 
[email protected]/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at 
[email protected]/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at 
[email protected]/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at 
app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:149)
        at 
app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:156)
        at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:572)
        at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:509)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
        at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$269/0x0000000100441c40.run(Unknown
 Source)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  25) Thread[id=573, name=CloserThread, state=WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/java.lang.Object.wait(Native Method)
        at [email protected]/java.lang.Object.wait(Object.java:328)
        at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2326)
  26) Thread[id=524, name=qtp1052695412-524, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:842)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:892)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  27) Thread[id=505, name=qtp1141594396-505, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:842)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:892)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  28) Thread[id=514, name=qtp89046542-514, state=TIMED_WAITING, 
group=TGRP-TestLTROnSolrCloud]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:842)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:892)
        at [email protected]/java.lang.Thread.run

[...truncated too long message...]

:contrib:ltr:test --tests 
"org.apache.solr.ltr.feature.TestValueFeature.testValueFeatureRequiredShouldReturn400StatusCode"
 -Ptests.jvms=4 -Ptests.haltonfailure=false 
-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -Ptests.seed=3E29518EECF3017D 
-Ptests.multiplier=2 -Ptests.badapples=false -Ptests.file.encoding=ISO-8859-1

  - org.apache.solr.ltr.model.TestNeuralNetworkModel.biasDimensionMismatchTest 
(:solr:contrib:ltr)
    Test output: 
/home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-main/solr/contrib/ltr/build/test-results/test/outputs/OUTPUT-org.apache.solr.ltr.model.TestNeuralNetworkModel.txt
    Reproduce with: gradlew :solr:contrib:ltr:test --tests 
"org.apache.solr.ltr.model.TestNeuralNetworkModel.biasDimensionMismatchTest" 
-Ptests.jvms=4 -Ptests.haltonfailure=false 
-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -Ptests.seed=3E29518EECF3017D 
-Ptests.multiplier=2 -Ptests.badapples=false -Ptests.file.encoding=ISO-8859-1

  - 
org.apache.solr.ltr.feature.TestFieldValueFeature.testIfADocumentDoesntHaveAFieldASetDefaultValueIsReturned
 (:solr:contrib:ltr)
    Test output: 
/home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-main/solr/contrib/ltr/build/test-results/test/outputs/OUTPUT-org.apache.solr.ltr.feature.TestFieldValueFeature.txt
    Reproduce with: gradlew :solr:contrib:ltr:test --tests 
"org.apache.solr.ltr.feature.TestFieldValueFeature.testIfADocumentDoesntHaveAFieldASetDefaultValueIsReturned"
 -Ptests.jvms=4 -Ptests.haltonfailure=false 
-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -Ptests.seed=3E29518EECF3017D 
-Ptests.multiplier=2 -Ptests.badapples=false -Ptests.file.encoding=ISO-8859-1

  - org.apache.solr.ltr.feature.TestValueFeature.testValueFeatureWithEmptyValue 
(:solr:contrib:ltr)
    Test output: 
/home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-main/solr/contrib/ltr/build/test-results/test/outputs/OUTPUT-org.apache.solr.ltr.feature.TestValueFeature.txt
    Reproduce with: gradlew :solr:contrib:ltr:test --tests 
"org.apache.solr.ltr.feature.TestValueFeature.testValueFeatureWithEmptyValue" 
-Ptests.jvms=4 -Ptests.haltonfailure=false 
-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -Ptests.seed=3E29518EECF3017D 
-Ptests.multiplier=2 -Ptests.badapples=false -Ptests.file.encoding=ISO-8859-1

  - org.apache.solr.ltr.feature.TestFieldValueFeature.testBooleanValue 
(:solr:contrib:ltr)
    Test output: 
/home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-main/solr/contrib/ltr/build/test-results/test/outputs/OUTPUT-org.apache.solr.ltr.feature.TestFieldValueFeature.txt
    Reproduce with: gradlew :solr:contrib:ltr:test --tests 
"org.apache.solr.ltr.feature.TestFieldValueFeature.testBooleanValue" 
-Ptests.jvms=4 -Ptests.haltonfailure=false 
-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -Ptests.seed=3E29518EECF3017D 
-Ptests.multiplier=2 -Ptests.badapples=false -Ptests.file.encoding=ISO-8859-1

  - 
org.apache.solr.ltr.feature.TestValueFeature.testValueFeatureImplicitlyNotRequiredShouldReturnOkStatusCode
 (:solr:contrib:ltr)
    Test output: 
/home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-main/solr/contrib/ltr/build/test-results/test/outputs/OUTPUT-org.apache.solr.ltr.feature.TestValueFeature.txt
    Reproduce with: gradlew :solr:contrib:ltr:test --tests 
"org.apache.solr.ltr.feature.TestValueFeature.testValueFeatureImplicitlyNotRequiredShouldReturnOkStatusCode"
 -Ptests.jvms=4 -Ptests.haltonfailure=false 
-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -Ptests.seed=3E29518EECF3017D 
-Ptests.multiplier=2 -Ptests.badapples=false -Ptests.file.encoding=ISO-8859-1

  - org.apache.solr.ltr.feature.TestRankingFeature.testRankingSolrFeature 
(:solr:contrib:ltr)
    Test output: 
/home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-main/solr/contrib/ltr/build/test-results/test/outputs/OUTPUT-org.apache.solr.ltr.feature.TestRankingFeature.txt
    Reproduce with: gradlew :solr:contrib:ltr:test --tests 
"org.apache.solr.ltr.feature.TestRankingFeature.testRankingSolrFeature" 
-Ptests.jvms=4 -Ptests.haltonfailure=false 
-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -Ptests.seed=3E29518EECF3017D 
-Ptests.multiplier=2 -Ptests.badapples=false -Ptests.file.encoding=ISO-8859-1

  - org.apache.solr.ltr.TestLTROnSolrCloud.testSimpleQuery (:solr:contrib:ltr)
    Test output: 
/home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-main/solr/contrib/ltr/build/test-results/test/outputs/OUTPUT-org.apache.solr.ltr.TestLTROnSolrCloud.txt
    Reproduce with: gradlew :solr:contrib:ltr:test --tests 
"org.apache.solr.ltr.TestLTROnSolrCloud.testSimpleQuery" -Ptests.jvms=4 
-Ptests.haltonfailure=false -Ptests.jvmargs=-XX:TieredStopAtLevel=1 
-Ptests.seed=3E29518EECF3017D -Ptests.multiplier=2 -Ptests.badapples=false 
-Ptests.file.encoding=ISO-8859-1

  - org.apache.solr.ltr.feature.TestRankingFeature.testParamsToMap 
(:solr:contrib:ltr)
    Test output: 
/home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-main/solr/contrib/ltr/build/test-results/test/outputs/OUTPUT-org.apache.solr.ltr.feature.TestRankingFeature.txt
    Reproduce with: gradlew :solr:contrib:ltr:test --tests 
"org.apache.solr.ltr.feature.TestRankingFeature.testParamsToMap" -Ptests.jvms=4 
-Ptests.haltonfailure=false -Ptests.jvmargs=-XX:TieredStopAtLevel=1 
-Ptests.seed=3E29518EECF3017D -Ptests.multiplier=2 -Ptests.badapples=false 
-Ptests.file.encoding=ISO-8859-1

  - org.apache.solr.ltr.feature.TestValueFeature.testParamsToMap 
(:solr:contrib:ltr)
    Test output: 
/home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-main/solr/contrib/ltr/build/test-results/test/outputs/OUTPUT-org.apache.solr.ltr.feature.TestValueFeature.txt
    Reproduce with: gradlew :solr:contrib:ltr:test --tests 
"org.apache.solr.ltr.feature.TestValueFeature.testParamsToMap" -Ptests.jvms=4 
-Ptests.haltonfailure=false -Ptests.jvmargs=-XX:TieredStopAtLevel=1 
-Ptests.seed=3E29518EECF3017D -Ptests.multiplier=2 -Ptests.badapples=false 
-Ptests.file.encoding=ISO-8859-1

  - 
org.apache.solr.ltr.feature.TestValueFeature.testRerankingWithConstantValueFeatureReplacesDocScore
 (:solr:contrib:ltr)
    Test output: 
/home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-main/solr/contrib/ltr/build/test-results/test/outputs/OUTPUT-org.apache.solr.ltr.feature.TestValueFeature.txt
    Reproduce with: gradlew :solr:contrib:ltr:test --tests 
"org.apache.solr.ltr.feature.TestValueFeature.testRerankingWithConstantValueFeatureReplacesDocScore"
 -Ptests.jvms=4 -Ptests.haltonfailure=false 
-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -Ptests.seed=3E29518EECF3017D 
-Ptests.multiplier=2 -Ptests.badapples=false -Ptests.file.encoding=ISO-8859-1

  - 
org.apache.solr.ltr.feature.TestUserTermScorereQDF.testUserTermScorerQWithDF 
(:solr:contrib:ltr)
    Test output: 
/home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-main/solr/contrib/ltr/build/test-results/test/outputs/OUTPUT-org.apache.solr.ltr.feature.TestUserTermScorereQDF.txt
    Reproduce with: gradlew :solr:contrib:ltr:test --tests 
"org.apache.solr.ltr.feature.TestUserTermScorereQDF.testUserTermScorerQWithDF" 
-Ptests.jvms=4 -Ptests.haltonfailure=false 
-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -Ptests.seed=3E29518EECF3017D 
-Ptests.multiplier=2 -Ptests.badapples=false -Ptests.file.encoding=ISO-8859-1

  - org.apache.solr.ltr.TestSelectiveWeightCreation.classMethod 
(:solr:contrib:ltr)
    Test output: 
/home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-main/solr/contrib/ltr/build/test-results/test/outputs/OUTPUT-org.apache.solr.ltr.TestSelectiveWeightCreation.txt
    Reproduce with: gradlew :solr:contrib:ltr:test --tests 
"org.apache.solr.ltr.TestSelectiveWeightCreation.classMethod" -Ptests.jvms=4 
-Ptests.haltonfailure=false -Ptests.jvmargs=-XX:TieredStopAtLevel=1 
-Ptests.seed=3E29518EECF3017D -Ptests.multiplier=2 -Ptests.badapples=false 
-Ptests.file.encoding=ISO-8859-1

  - org.apache.solr.ltr.TestLTROnSolrCloud.classMethod (:solr:contrib:ltr)
    Test output: 
/home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-main/solr/contrib/ltr/build/test-results/test/outputs/OUTPUT-org.apache.solr.ltr.TestLTROnSolrCloud.txt
    Reproduce with: gradlew :solr:contrib:ltr:test --tests 
"org.apache.solr.ltr.TestLTROnSolrCloud.classMethod" -Ptests.jvms=4 
-Ptests.haltonfailure=false -Ptests.jvmargs=-XX:TieredStopAtLevel=1 
-Ptests.seed=3E29518EECF3017D -Ptests.multiplier=2 -Ptests.badapples=false 
-Ptests.file.encoding=ISO-8859-1

  - org.apache.solr.ltr.TestLTROnSolrCloud.classMethod (:solr:contrib:ltr)
    Test output: 
/home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-main/solr/contrib/ltr/build/test-results/test/outputs/OUTPUT-org.apache.solr.ltr.TestLTROnSolrCloud.txt
    Reproduce with: gradlew :solr:contrib:ltr:test --tests 
"org.apache.solr.ltr.TestLTROnSolrCloud.classMethod" -Ptests.jvms=4 
-Ptests.haltonfailure=false -Ptests.jvmargs=-XX:TieredStopAtLevel=1 
-Ptests.seed=3E29518EECF3017D -Ptests.multiplier=2 -Ptests.badapples=false 
-Ptests.file.encoding=ISO-8859-1


FAILURE: Build failed with an exception.

* Where:
Script 
'/home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-main/gradle/validation/rat-sources.gradle'
 line: 199

* What went wrong:
Execution failed for task ':solr:core:rat'.
> Found 1 file(s) with errors:
    - Unknown license: 
/home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-main/solr/core/src/java/org/apache/solr/core/OverlaidConfigNode.java

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with 
Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See 
https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 48m 0s
296 actionable tasks: 296 executed
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to