[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk-11.0.2) - Build # 5211 - Still Unstable!

Policeman Jenkins Server Thu, 20 Jun 2019 15:22:56 -0700

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/5211/
Java: 64bit/jdk-11.0.2 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC


12 tests failed.
FAILED:  org.apache.solr.cloud.TestStressInPlaceUpdates.stressTest

Error Message:
Number of replicas in the state does not match what we set:2 vs 3

Stack Trace:
org.apache.solr.cloud.ZkController$NotInClusterStateException: Number of 
replicas in the state does not match what we set:2 vs 3
        at 
__randomizedtesting.SeedInfo.seed([EE7200379ED406F3:8514DF9AA001D209]:0)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForActiveReplicaCount(AbstractFullDistribZkTestBase.java:603)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:552)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:358)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1080)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1054)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:834)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.TestStressInPlaceUpdates

Error Message:
41 threads leaked from SUITE scope at 
org.apache.solr.cloud.TestStressInPlaceUpdates:     1) Thread[id=179, 
name=qtp1473955222-179, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
         at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
         at [email protected]/java.lang.Thread.run(Thread.java:834)    2) 
Thread[id=170, name=MetricsHistoryHandler-50-thread-1, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
         at [email protected]/java.lang.Thread.run(Thread.java:834)    3) 
Thread[id=169, name=SolrRrdBackendFactory-49-thread-1, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
         at [email protected]/java.lang.Thread.run(Thread.java:834)    4) 
Thread[id=111, name=qtp1259397275-111, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
         at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
         at [email protected]/java.lang.Thread.run(Thread.java:834)    5) 
Thread[id=135, name=SolrRrdBackendFactory-40-thread-1, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
         at [email protected]/java.lang.Thread.run(Thread.java:834)    6) 
Thread[id=140, name=qtp578218321-140, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
         at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
         at [email protected]/java.lang.Thread.run(Thread.java:834)    7) 
Thread[id=181, name=qtp1473955222-181, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
         at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
         at [email protected]/java.lang.Thread.run(Thread.java:834)    8) 
Thread[id=175, 
name=qtp1473955222-175-acceptor-۰@5eb7c69b-ServerConnector@45335af8{ssl,[ssl, 
alpn, http/1.1, h2]}{127.0.0.1:۶۳۳۳۳}, state=RUNNABLE, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)      
   at 
[email protected]/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
         at 
[email protected]/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
         at 
app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385)  
       at 
app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:648)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917)
         at [email protected]/java.lang.Thread.run(Thread.java:834)    9) 
Thread[id=150, name=Thread-27, state=WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/java.lang.Object.wait(Native Method)         at 
[email protected]/java.lang.Object.wait(Object.java:328)         at 
app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2018)   10) 
Thread[id=117, name=Session-HouseKeeper-3817191e, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
         at [email protected]/java.lang.Thread.run(Thread.java:834)   11) 
Thread[id=136, name=MetricsHistoryHandler-41-thread-1, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
         at [email protected]/java.lang.Thread.run(Thread.java:834)   12) 
Thread[id=142, name=qtp578218321-142, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
         at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
         at [email protected]/java.lang.Thread.run(Thread.java:834)   13) 
Thread[id=112, name=qtp1259397275-112, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
         at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
         at [email protected]/java.lang.Thread.run(Thread.java:834)   14) 
Thread[id=145, name=qtp578218321-145, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
         at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
         at [email protected]/java.lang.Thread.run(Thread.java:834)   15) 
Thread[id=202, name=SolrRrdBackendFactory-58-thread-1, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
         at [email protected]/java.lang.Thread.run(Thread.java:834)   16) 
Thread[id=114, name=qtp1259397275-114, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
         at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
         at [email protected]/java.lang.Thread.run(Thread.java:834)   17) 
Thread[id=176, name=qtp1473955222-176, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
         at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
         at [email protected]/java.lang.Thread.run(Thread.java:834)   18) 
Thread[id=143, name=qtp578218321-143, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
         at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
         at [email protected]/java.lang.Thread.run(Thread.java:834)   19) 
Thread[id=182, name=Session-HouseKeeper-4929f3a8, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
         at [email protected]/java.lang.Thread.run(Thread.java:834)   20) 
Thread[id=116, name=qtp1259397275-116, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
         at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
         at [email protected]/java.lang.Thread.run(Thread.java:834)   21) 
Thread[id=171, name=Thread-28, state=WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/java.lang.Object.wait(Native Method)         at 
[email protected]/java.lang.Object.wait(Object.java:328)         at 
app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2018)   22) 
Thread[id=178, name=qtp1473955222-178, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
         at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
         at [email protected]/java.lang.Thread.run(Thread.java:834)   23) 
Thread[id=174, name=qtp1473955222-174, state=RUNNABLE, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/sun.nio.ch.KQueue.poll(Native Method)         at 
[email protected]/sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:122)
         at 
[email protected]/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) 
        at 
[email protected]/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)         
at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:464)
         at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:401)
         at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
         at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
         at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
         at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
         at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$187/0x00000001418c4c58.run(Unknown
 Source)         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917)
         at [email protected]/java.lang.Thread.run(Thread.java:834)   24) 
Thread[id=214, name=Connector-Scheduler-634be36a, state=WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
         at [email protected]/java.lang.Thread.run(Thread.java:834)   25) 
Thread[id=299, name=SolrRrdBackendFactory-49-thread-2, state=WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1177)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
         at [email protected]/java.lang.Thread.run(Thread.java:834)   26) 
Thread[id=109, name=qtp1259397275-109, state=RUNNABLE, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/sun.nio.ch.KQueue.poll(Native Method)         at 
[email protected]/sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:122)
         at 
[email protected]/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) 
        at 
[email protected]/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)         
at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:464)
         at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:401)
         at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
         at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
         at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
         at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
         at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$187/0x00000001418c4c58.run(Unknown
 Source)         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917)
         at [email protected]/java.lang.Thread.run(Thread.java:834)   27) 
Thread[id=113, name=qtp1259397275-113, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
         at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
         at [email protected]/java.lang.Thread.run(Thread.java:834)   28) 
Thread[id=300, name=SolrRrdBackendFactory-58-thread-2, state=WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1177)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
         at [email protected]/java.lang.Thread.run(Thread.java:834)   29) 
Thread[id=138, name=qtp578218321-138, state=RUNNABLE, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/sun.nio.ch.KQueue.poll(Native Method)         at 
[email protected]/sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:122)
         at 
[email protected]/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) 
        at 
[email protected]/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)         
at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:464)
         at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:401)
         at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
         at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
         at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
         at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
         at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$187/0x00000001418c4c58.run(Unknown
 Source)         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917)
         at [email protected]/java.lang.Thread.run(Thread.java:834)   30) 
Thread[id=177, name=qtp1473955222-177, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
         at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
         at [email protected]/java.lang.Thread.run(Thread.java:834)   31) 
Thread[id=144, name=qtp578218321-144, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
         at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
         at [email protected]/java.lang.Thread.run(Thread.java:834)   32) 
Thread[id=298, name=SolrRrdBackendFactory-40-thread-2, state=WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1177)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
         at [email protected]/java.lang.Thread.run(Thread.java:834)   33) 
Thread[id=115, name=qtp1259397275-115, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
         at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
         at [email protected]/java.lang.Thread.run(Thread.java:834)   34) 
Thread[id=204, name=Thread-37, state=WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/java.lang.Object.wait(Native Method)         at 
[email protected]/java.lang.Object.wait(Object.java:328)         at 
app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2018)   35) 
Thread[id=139, 
name=qtp578218321-139-acceptor-۰@925f0eb-ServerConnector@634be36a{ssl,[ssl, 
alpn, http/1.1, h2]}{127.0.0.1:۶۳۳۲۶}, state=RUNNABLE, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)      
   at 
[email protected]/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
         at 
[email protected]/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
         at 
app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385)  
       at 
app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:648)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917)
         at [email protected]/java.lang.Thread.run(Thread.java:834)   36) 
Thread[id=203, name=MetricsHistoryHandler-59-thread-1, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
         at [email protected]/java.lang.Thread.run(Thread.java:834)   37) 
Thread[id=146, name=Session-HouseKeeper-7e8d59ef, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
         at [email protected]/java.lang.Thread.run(Thread.java:834)   38) 
Thread[id=141, name=qtp578218321-141, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
         at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
         at [email protected]/java.lang.Thread.run(Thread.java:834)   39) 
Thread[id=180, name=qtp1473955222-180, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
         at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
         at [email protected]/java.lang.Thread.run(Thread.java:834)   40) 
Thread[id=224, name=Connector-Scheduler-a2deaa0, state=WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
         at [email protected]/java.lang.Thread.run(Thread.java:834)   41) 
Thread[id=110, 
name=qtp1259397275-110-acceptor-۰@3382af2f-ServerConnector@a2deaa0{ssl,[ssl, 
alpn, http/1.1, h2]}{127.0.0.1:۶۳۳۱۳}, state=RUNNABLE, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)      
   at 
[email protected]/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
         at 
[email protected]/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
         at 
app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385)  
       at 
app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:648)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917)
         at [email protected]/java.lang.Thread.run(Thread.java:834)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 41 threads leaked from 
SUITE scope at org.apache.solr.cloud.TestStressInPlaceUpdates: 
   1) Thread[id=179, name=qtp1473955222-179, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
   2) Thread[id=170, name=MetricsHistoryHandler-50-thread-1, 
state=TIMED_WAITING, group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
   3) Thread[id=169, name=SolrRrdBackendFactory-49-thread-1, 
state=TIMED_WAITING, group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
   4) Thread[id=111, name=qtp1259397275-111, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
   5) Thread[id=135, name=SolrRrdBackendFactory-40-thread-1, 
state=TIMED_WAITING, group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
   6) Thread[id=140, name=qtp578218321-140, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
   7) Thread[id=181, name=qtp1473955222-181, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
   8) Thread[id=175, 
name=qtp1473955222-175-acceptor-۰@5eb7c69b-ServerConnector@45335af8{ssl,[ssl, 
alpn, http/1.1, h2]}{127.0.0.1:۶۳۳۳۳}, state=RUNNABLE, 
group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/sun.nio.ch.ServerSocketChannelImpl.accept0(Native 
Method)
        at 
[email protected]/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
        at 
[email protected]/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
        at 
app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385)
        at 
app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:648)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
   9) Thread[id=150, name=Thread-27, state=WAITING, 
group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/java.lang.Object.wait(Native Method)
        at [email protected]/java.lang.Object.wait(Object.java:328)
        at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2018)
  10) Thread[id=117, name=Session-HouseKeeper-3817191e, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  11) Thread[id=136, name=MetricsHistoryHandler-41-thread-1, 
state=TIMED_WAITING, group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  12) Thread[id=142, name=qtp578218321-142, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  13) Thread[id=112, name=qtp1259397275-112, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  14) Thread[id=145, name=qtp578218321-145, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  15) Thread[id=202, name=SolrRrdBackendFactory-58-thread-1, 
state=TIMED_WAITING, group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  16) Thread[id=114, name=qtp1259397275-114, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  17) Thread[id=176, name=qtp1473955222-176, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  18) Thread[id=143, name=qtp578218321-143, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  19) Thread[id=182, name=Session-HouseKeeper-4929f3a8, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  20) Thread[id=116, name=qtp1259397275-116, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  21) Thread[id=171, name=Thread-28, state=WAITING, 
group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/java.lang.Object.wait(Native Method)
        at [email protected]/java.lang.Object.wait(Object.java:328)
        at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2018)
  22) Thread[id=178, name=qtp1473955222-178, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  23) Thread[id=174, name=qtp1473955222-174, state=RUNNABLE, 
group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/sun.nio.ch.KQueue.poll(Native Method)
        at 
[email protected]/sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:122)
        at 
[email protected]/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at 
[email protected]/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:464)
        at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:401)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$187/0x00000001418c4c58.run(Unknown
 Source)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  24) Thread[id=214, name=Connector-Scheduler-634be36a, state=WAITING, 
group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  25) Thread[id=299, name=SolrRrdBackendFactory-49-thread-2, state=WAITING, 
group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1177)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  26) Thread[id=109, name=qtp1259397275-109, state=RUNNABLE, 
group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/sun.nio.ch.KQueue.poll(Native Method)
        at 
[email protected]/sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:122)
        at 
[email protected]/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at 
[email protected]/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:464)
        at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:401)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$187/0x00000001418c4c58.run(Unknown
 Source)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  27) Thread[id=113, name=qtp1259397275-113, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  28) Thread[id=300, name=SolrRrdBackendFactory-58-thread-2, state=WAITING, 
group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1177)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  29) Thread[id=138, name=qtp578218321-138, state=RUNNABLE, 
group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/sun.nio.ch.KQueue.poll(Native Method)
        at 
[email protected]/sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:122)
        at 
[email protected]/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at 
[email protected]/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:464)
        at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:401)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at 
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
        at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$187/0x00000001418c4c58.run(Unknown
 Source)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  30) Thread[id=177, name=qtp1473955222-177, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  31) Thread[id=144, name=qtp578218321-144, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  32) Thread[id=298, name=SolrRrdBackendFactory-40-thread-2, state=WAITING, 
group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1177)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  33) Thread[id=115, name=qtp1259397275-115, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  34) Thread[id=204, name=Thread-37, state=WAITING, 
group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/java.lang.Object.wait(Native Method)
        at [email protected]/java.lang.Object.wait(Object.java:328)
        at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2018)
  35) Thread[id=139, 
name=qtp578218321-139-acceptor-۰@925f0eb-ServerConnector@634be36a{ssl,[ssl, 
alpn, http/1.1, h2]}{127.0.0.1:۶۳۳۲۶}, state=RUNNABLE, 
group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/sun.nio.ch.ServerSocketChannelImpl.accept0(Native 
Method)
        at 
[email protected]/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
        at 
[email protected]/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
        at 
app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385)
        at 
app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:648)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  36) Thread[id=203, name=MetricsHistoryHandler-59-thread-1, 
state=TIMED_WAITING, group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  37) Thread[id=146, name=Session-HouseKeeper-7e8d59ef, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  38) Thread[id=141, name=qtp578218321-141, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  39) Thread[id=180, name=qtp1473955222-180, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  40) Thread[id=224, name=Connector-Scheduler-a2deaa0, state=WAITING, 
group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
  41) Thread[id=110, 
name=qtp1259397275-110-acceptor-۰@3382af2f-ServerConnector@a2deaa0{ssl,[ssl, 
alpn, http/1.1, h2]}{127.0.0.1:۶۳۳۱۳}, state=RUNNABLE, 
group=TGRP-TestStressInPlaceUpdates]
        at [email protected]/sun.nio.ch.ServerSocketChannelImpl.accept0(Native 
Method)
        at 
[email protected]/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
        at 
[email protected]/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
        at 
app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385)
        at 
app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:648)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781)
        at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
        at __randomizedtesting.SeedInfo.seed([EE7200379ED406F3]:0)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.TestStressInPlaceUpdates

Error Message:
There are still zombie threads that couldn't be terminated:    1) 
Thread[id=179, name=qtp1473955222-179, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
         at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
         at [email protected]/java.lang.Thread.run(Thread.java:834)    2) 
Thread[id=170, name=MetricsHistoryHandler-50-thread-1, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
         at [email protected]/java.lang.Thread.run(Thread.java:834)    3) 
Thread[id=169, name=SolrRrdBackendFactory-49-thread-1, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
         at [email protected]/java.lang.Thread.run(Thread.java:834)    4) 
Thread[id=111, name=qtp1259397275-111, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
         at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
         at [email protected]/java.lang.Thread.run(Thread.java:834)    5) 
Thread[id=135, name=SolrRrdBackendFactory-40-thread-1, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
         at [email protected]/java.lang.Thread.run(Thread.java:834)    6) 
Thread[id=140, name=qtp578218321-140, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
         at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
         at [email protected]/java.lang.Thread.run(Thread.java:834)    7) 
Thread[id=181, name=qtp1473955222-181, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
         at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
         at [email protected]/java.lang.Thread.run(Thread.java:834)    8) 
Thread[id=175, name=qtp1473955222-175, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
         at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
         at [email protected]/java.lang.Thread.run(Thread.java:834)    9) 
Thread[id=150, name=Thread-27, state=WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/java.lang.Object.wait(Native Method)         at 
[email protected]/java.lang.Object.wait(Object.java:328)         at 
app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2018)   10) 
Thread[id=117, name=Session-HouseKeeper-3817191e, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
         at [email protected]/java.lang.Thread.run(Thread.java:834)   11) 
Thread[id=136, name=MetricsHistoryHandler-41-thread-1, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
         at [email protected]/java.lang.Thread.run(Thread.java:834)   12) 
Thread[id=142, name=qtp578218321-142, state=TIMED_WAITING, 
group=TGRP-TestStressInPlaceUpdates]         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll

[...truncated too long message...]

rt.java:234)
   [junit4]    >         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]    >         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
   [junit4]    >         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
   [junit4]    >         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
   [junit4]    >         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
   [junit4]    >         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
   [junit4]    >         at 
[email protected]/java.lang.Thread.run(Thread.java:834)
   [junit4]    >   37) Thread[id=146, name=Session-HouseKeeper-7e8d59ef, 
state=TIMED_WAITING, group=TGRP-TestStressInPlaceUpdates]
   [junit4]    >         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]    >         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]    >         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
   [junit4]    >         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
   [junit4]    >         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
   [junit4]    >         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
   [junit4]    >         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
   [junit4]    >         at 
[email protected]/java.lang.Thread.run(Thread.java:834)
   [junit4]    >   38) Thread[id=141, name=qtp578218321-141, 
state=TIMED_WAITING, group=TGRP-TestStressInPlaceUpdates]
   [junit4]    >         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]    >         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]    >         at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
   [junit4]    >         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
   [junit4]    >         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
   [junit4]    >         at 
[email protected]/java.lang.Thread.run(Thread.java:834)
   [junit4]    >   39) Thread[id=180, name=qtp1473955222-180, 
state=TIMED_WAITING, group=TGRP-TestStressInPlaceUpdates]
   [junit4]    >         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]    >         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]    >         at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
   [junit4]    >         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
   [junit4]    >         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
   [junit4]    >         at 
[email protected]/java.lang.Thread.run(Thread.java:834)
   [junit4]    >   40) Thread[id=224, name=Connector-Scheduler-a2deaa0, 
state=WAITING, group=TGRP-TestStressInPlaceUpdates]
   [junit4]    >         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
[email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
   [junit4]    >         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
   [junit4]    >         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)
   [junit4]    >         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
   [junit4]    >         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
   [junit4]    >         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
   [junit4]    >         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
   [junit4]    >         at 
[email protected]/java.lang.Thread.run(Thread.java:834)
   [junit4]    >   41) Thread[id=110, name=qtp1259397275-110, 
state=TIMED_WAITING, group=TGRP-TestStressInPlaceUpdates]
   [junit4]    >         at 
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]    >         at 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]    >         at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
   [junit4]    >         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)
   [junit4]    >         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889)
   [junit4]    >         at 
[email protected]/java.lang.Thread.run(Thread.java:834)
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([EE7200379ED406F3]:0)
   [junit4] Completed [8/10 (2!)] on J0 in 372.16s, 1 test, 3 errors <<< 
FAILURES!
   [junit4] 
   [junit4] Suite: org.apache.solr.cloud.TestStressInPlaceUpdates
   [junit4] Completed [9/10 (2!)] on J0 in 0.01s, 1 test, 1 skipped
   [junit4] 
   [junit4] Suite: org.apache.solr.update.TestInPlaceUpdatesDistrib
   [junit4] Completed [10/10 (2!)] on J0 in 0.01s, 1 test, 1 skipped
   [junit4] 
   [junit4] JVM J0: stderr was not empty, see: 
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/temp/junit4-J0-20190620_220938_7162062755346620871850.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was 
deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----
   [junit4] 
   [junit4] Tests with failures [seed: EE7200379ED406F3]:
   [junit4]   - org.apache.solr.cloud.TestStressInPlaceUpdates.stressTest
   [junit4]   - org.apache.solr.cloud.TestStressInPlaceUpdates.stressTest
   [junit4]   - org.apache.solr.cloud.TestStressInPlaceUpdates (suite)
   [junit4]   - org.apache.solr.cloud.TestStressInPlaceUpdates (suite)
   [junit4] 
   [junit4] 
   [junit4] JVM J0:     1.34 ..   375.51 =   374.17s
   [junit4] JVM J1:     1.33 ..   228.02 =   226.69s
   [junit4] Execution time total: 6 minutes 15 seconds
   [junit4] Tests summary: 10 suites, 10 tests, 4 suite-level errors, 2 errors, 
8 ignored

BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:1492:
 The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:1019:
 There were test failures: 10 suites, 10 tests, 4 suite-level errors, 2 errors, 
8 ignored [seed: EE7200379ED406F3]

Total time: 6 minutes 19 seconds

[repro] Setting last failure code to 256

[repro] Failures:
[repro]   0/5 failed: org.apache.solr.update.TestInPlaceUpdatesDistrib
[repro]   2/5 failed: org.apache.solr.cloud.TestStressInPlaceUpdates
[repro] Exiting with code 256
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Archiving artifacts
Setting 
ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting 
ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
No credentials specified
Setting 
ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all 
warnings.
<Git Blamer> GIT_COMMIT=df3011a5391f548db9db34abc7d5e158d582f871, 
workspace=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX
[WARNINGS] Computing warning deltas based on reference build #5210
Recording test results
Setting 
ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting 
ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk-11.0.2) - Build # 5211 - Still Unstable!

Reply via email to