Build: https://jenkins.thetaphi.de/job/Lucene-Solr-http2-Linux/56/
Java: 64bit/jdk-11 -XX:-UseCompressedOops -XX:+UseParallelGC
19 tests failed.
FAILED:
junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
Error Message:
74 threads leaked from SUITE scope at
org.apache.solr.cloud.ChaosMonkeySafeLeaderTest: 1) Thread[id=1071,
name=qtp154888852-1071, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
at java.base@11/sun.nio.ch.EPoll.wait(Native Method) at
java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at
java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown
Source) at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base@11/java.lang.Thread.run(Thread.java:834) 2)
Thread[id=1001, name=qtp1645118601-1001, state=RUNNABLE,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/sun.nio.ch.EPoll.wait(Native Method) at
java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at
java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown
Source) at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base@11/java.lang.Thread.run(Thread.java:834) 3)
Thread[id=1279, name=updateExecutor-188-thread-2, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
at
java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
at
java.base@11/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834) 4)
Thread[id=1253, name=Scheduler-2089236759, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834) 5)
Thread[id=977, name=qtp2079431130-977, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
at java.base@11/java.lang.Thread.run(Thread.java:834) 6)
Thread[id=1037, name=qtp1242769164-1037, state=RUNNABLE,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/sun.nio.ch.EPoll.wait(Native Method) at
java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at
java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown
Source) at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base@11/java.lang.Thread.run(Thread.java:834) 7)
Thread[id=1213, name=coreContainerWorkExecutor-213-thread-1,
state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
at
java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
at
java.base@11/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834) 8)
Thread[id=1255, name=Scheduler-1208377519, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834) 9)
Thread[id=923, name=qtp66645842-923, state=RUNNABLE,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/sun.nio.ch.EPoll.wait(Native Method) at
java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at
java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown
Source) at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base@11/java.lang.Thread.run(Thread.java:834) 10)
Thread[id=1002,
name=qtp1645118601-1002-acceptor-0@cbb0765-ServerConnector@4695d09b{ssl,[ssl,
alpn, http/1.1, h2]}{127.0.0.1:41353}, state=RUNNABLE,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
at
java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
at
app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385)
at
app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:648)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base@11/java.lang.Thread.run(Thread.java:834) 11)
Thread[id=1229, name=SolrRrdBackendFactory-242-thread-1, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834) 12)
Thread[id=1275, name=updateExecutor-231-thread-1, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
at
java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
at
java.base@11/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834) 13)
Thread[id=924, name=qtp66645842-924, state=RUNNABLE,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/sun.nio.ch.EPoll.wait(Native Method) at
java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at
java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown
Source) at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base@11/java.lang.Thread.run(Thread.java:834) 14)
Thread[id=1000, name=qtp1645118601-1000, state=RUNNABLE,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/sun.nio.ch.EPoll.wait(Native Method) at
java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at
java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown
Source) at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base@11/java.lang.Thread.run(Thread.java:834) 15)
Thread[id=928, name=qtp66645842-928, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
at java.base@11/java.lang.Thread.run(Thread.java:834) 16)
Thread[id=1045,
name=qtp1242769164-1045-acceptor-0@74e7ce15-ServerConnector@7b69117d{ssl,[ssl,
alpn, http/1.1, h2]}{127.0.0.1:44499}, state=RUNNABLE,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
at
java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
at
app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385)
at
app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:648)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base@11/java.lang.Thread.run(Thread.java:834) 17)
Thread[id=972, name=qtp2079431130-972, state=RUNNABLE,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/sun.nio.ch.EPoll.wait(Native Method) at
java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at
java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown
Source) at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base@11/java.lang.Thread.run(Thread.java:834) 18)
Thread[id=999, name=qtp1645118601-999, state=RUNNABLE,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/sun.nio.ch.EPoll.wait(Native Method) at
java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at
java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown
Source) at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base@11/java.lang.Thread.run(Thread.java:834) 19)
Thread[id=998, name=qtp1645118601-998, state=RUNNABLE,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/sun.nio.ch.EPoll.wait(Native Method) at
java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at
java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown
Source) at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base@11/java.lang.Thread.run(Thread.java:834) 20)
Thread[id=1077, name=qtp154888852-1077, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
at java.base@11/java.lang.Thread.run(Thread.java:834) 21)
Thread[id=995, name=coreContainerWorkExecutor-202-thread-1,
state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
at
java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
at
java.base@11/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834) 22)
Thread[id=1234, name=coreContainerWorkExecutor-218-thread-1,
state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
at
java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
at
java.base@11/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834) 23)
Thread[id=1232, name=coreContainerWorkExecutor-227-thread-1,
state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
at
java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
at
java.base@11/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834) 24)
Thread[id=974,
name=qtp2079431130-974-acceptor-0@14fbca69-ServerConnector@5fde07f3{ssl,[ssl,
alpn, http/1.1, h2]}{127.0.0.1:43939}, state=RUNNABLE,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
at
java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
at
app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385)
at
app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:648)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base@11/java.lang.Thread.run(Thread.java:834) 25)
Thread[id=1075,
name=qtp154888852-1075-acceptor-0@5d6ed29a-ServerConnector@54088673{ssl,[ssl,
alpn, http/1.1, h2]}{127.0.0.1:43629}, state=RUNNABLE,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
at
java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
at
app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385)
at
app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:648)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base@11/java.lang.Thread.run(Thread.java:834) 26)
Thread[id=1006, name=Scheduler-916823175, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834) 27)
Thread[id=1043, name=qtp1242769164-1043, state=RUNNABLE,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/sun.nio.ch.EPoll.wait(Native Method) at
java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at
java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown
Source) at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base@11/java.lang.Thread.run(Thread.java:834) 28)
Thread[id=931, name=Scheduler-1974536469, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834) 29)
Thread[id=980, name=Scheduler-1939903640, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834) 30)
Thread[id=1090, name=coreContainerWorkExecutor-207-thread-1,
state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
at
java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
at
java.base@11/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834) 31)
Thread[id=1072, name=qtp154888852-1072, state=RUNNABLE,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/sun.nio.ch.EPoll.wait(Native Method) at
java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at
java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown
Source) at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base@11/java.lang.Thread.run(Thread.java:834) 32)
Thread[id=1256, name=Scheduler-986485729, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834) 33)
Thread[id=993, name=MetricsHistoryHandler-210-thread-1, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834) 34)
Thread[id=1257, name=Scheduler-187575607, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834) 35)
Thread[id=1005, name=qtp1645118601-1005, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
at java.base@11/java.lang.Thread.run(Thread.java:834) 36)
Thread[id=926, name=qtp66645842-926, state=RUNNABLE,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/sun.nio.ch.EPoll.wait(Native Method) at
java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at
java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown
Source) at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base@11/java.lang.Thread.run(Thread.java:834) 37)
Thread[id=1073, name=qtp154888852-1073, state=RUNNABLE,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/sun.nio.ch.EPoll.wait(Native Method) at
java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at
java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown
Source) at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base@11/java.lang.Thread.run(Thread.java:834) 38)
Thread[id=1079, name=qtp154888852-1079, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
at java.base@11/java.lang.Thread.run(Thread.java:834) 39)
Thread[id=994, name=Thread-532, state=WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/java.lang.Object.wait(Native Method) at
java.base@11/java.lang.Object.wait(Object.java:328) at
app//org.apache.solr.core.CloserThread.run(CoreContainer.java:1902) 40)
Thread[id=1233, name=Thread-688, state=WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/java.lang.Object.wait(Native Method) at
java.base@11/java.lang.Object.wait(Object.java:328) at
app//org.apache.solr.core.CloserThread.run(CoreContainer.java:1902) 41)
Thread[id=973, name=qtp2079431130-973, state=RUNNABLE,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/sun.nio.ch.EPoll.wait(Native Method) at
java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at
java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown
Source) at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base@11/java.lang.Thread.run(Thread.java:834) 42)
Thread[id=929, name=qtp66645842-929, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
at java.base@11/java.lang.Thread.run(Thread.java:834) 43)
Thread[id=1003, name=qtp1645118601-1003, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
at java.base@11/java.lang.Thread.run(Thread.java:834) 44)
Thread[id=1039, name=qtp1242769164-1039, state=RUNNABLE,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/sun.nio.ch.EPoll.wait(Native Method) at
java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at
java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown
Source) at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base@11/java.lang.Thread.run(Thread.java:834) 45)
Thread[id=1004, name=qtp1645118601-1004, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
at java.base@11/java.lang.Thread.run(Thread.java:834) 46)
Thread[id=1041, name=qtp1242769164-1041, state=RUNNABLE,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/sun.nio.ch.EPoll.wait(Native Method) at
java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at
java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown
Source) at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base@11/java.lang.Thread.run(Thread.java:834) 47)
Thread[id=927,
name=qtp66645842-927-acceptor-0@58f24658-ServerConnector@3b602680{ssl,[ssl,
alpn, http/1.1, h2]}{127.0.0.1:43429}, state=RUNNABLE,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
at
java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
at
app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385)
at
app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:648)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base@11/java.lang.Thread.run(Thread.java:834) 48)
Thread[id=1086, name=Scheduler-934184420, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834) 49)
Thread[id=992, name=SolrRrdBackendFactory-209-thread-1, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834) 50)
Thread[id=1047, name=qtp1242769164-1047, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
at java.base@11/java.lang.Thread.run(Thread.java:834) 51)
Thread[id=1051, name=qtp1242769164-1051, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
at java.base@11/java.lang.Thread.run(Thread.java:834) 52)
Thread[id=1231, name=Thread-687, state=WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/java.lang.Object.wait(Native Method) at
java.base@11/java.lang.Object.wait(Object.java:328) at
app//org.apache.solr.core.CloserThread.run(CoreContainer.java:1902) 53)
Thread[id=930, name=qtp66645842-930, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
at java.base@11/java.lang.Thread.run(Thread.java:834) 54)
Thread[id=1225, name=SolrRrdBackendFactory-240-thread-1, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834) 55)
Thread[id=1179, name=SolrRrdBackendFactory-235-thread-1, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834) 56)
Thread[id=1053, name=Scheduler-1069203947, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834) 57)
Thread[id=975, name=qtp2079431130-975, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
at java.base@11/java.lang.Thread.run(Thread.java:834) 58)
Thread[id=1074, name=qtp154888852-1074, state=RUNNABLE,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/sun.nio.ch.EPoll.wait(Native Method) at
java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at
java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown
Source) at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base@11/java.lang.Thread.run(Thread.java:834) 59)
Thread[id=1089, name=Thread-598, state=WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/java.lang.Object.wait(Native Method) at
java.base@11/java.lang.Object.wait(Object.java:328) at
app//org.apache.solr.core.CloserThread.run(CoreContainer.java:1902) 60)
Thread[id=1228, name=MetricsHistoryHandler-241-thread-1, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834) 61)
Thread[id=925, name=qtp66645842-925, state=RUNNABLE,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/sun.nio.ch.EPoll.wait(Native Method) at
java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at
java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown
Source) at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base@11/java.lang.Thread.run(Thread.java:834) 62)
Thread[id=1278, name=updateExecutor-188-thread-1, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
at
java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
at
java.base@11/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834) 63)
Thread[id=1049, name=qtp1242769164-1049, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
at java.base@11/java.lang.Thread.run(Thread.java:834) 64)
Thread[id=1230, name=MetricsHistoryHandler-243-thread-1, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834) 65)
Thread[id=1254, name=Scheduler-1120265577, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834) 66)
Thread[id=1081, name=qtp154888852-1081, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
at java.base@11/java.lang.Thread.run(Thread.java:834) 67)
Thread[id=1068, name=SolrRrdBackendFactory-220-thread-1, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834) 68)
Thread[id=976, name=qtp2079431130-976, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
at java.base@11/java.lang.Thread.run(Thread.java:834) 69)
Thread[id=1069, name=MetricsHistoryHandler-221-thread-1, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834) 70)
Thread[id=1276, name=recoveryExecutor-240-thread-1, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
at
java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
at
java.base@11/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834) 71)
Thread[id=971, name=qtp2079431130-971, state=RUNNABLE,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/sun.nio.ch.EPoll.wait(Native Method) at
java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at
java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown
Source) at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base@11/java.lang.Thread.run(Thread.java:834) 72)
Thread[id=1181, name=MetricsHistoryHandler-236-thread-1, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834) 73)
Thread[id=1208, name=Thread-686, state=WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/java.lang.Object.wait(Native Method) at
java.base@11/java.lang.Object.wait(Object.java:328) at
app//org.apache.solr.core.CloserThread.run(CoreContainer.java:1902) 74)
Thread[id=970, name=qtp2079431130-970, state=RUNNABLE,
group=TGRP-ChaosMonkeySafeLeaderTest] at
java.base@11/sun.nio.ch.EPoll.wait(Native Method) at
java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at
java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown
Source) at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base@11/java.lang.Thread.run(Thread.java:834)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 74 threads leaked from
SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest:
1) Thread[id=1071, name=qtp154888852-1071, state=RUNNABLE,
group=TGRP-ChaosMonkeySafeLeaderTest]
at java.base@11/sun.nio.ch.EPoll.wait(Native Method)
at
java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at
java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown
Source)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base@11/java.lang.Thread.run(Thread.java:834)
2) Thread[id=1001, name=qtp1645118601-1001, state=RUNNABLE,
group=TGRP-ChaosMonkeySafeLeaderTest]
at java.base@11/sun.nio.ch.EPoll.wait(Native Method)
at
java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at
java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown
Source)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base@11/java.lang.Thread.run(Thread.java:834)
3) Thread[id=1279, name=updateExecutor-188-thread-2, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest]
at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
at
java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
at
java.base@11/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834)
4) Thread[id=1253, name=Scheduler-2089236759, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest]
at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834)
5) Thread[id=977, name=qtp2079431130-977, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest]
at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
at java.base@11/java.lang.Thread.run(Thread.java:834)
6) Thread[id=1037, name=qtp1242769164-1037, state=RUNNABLE,
group=TGRP-ChaosMonkeySafeLeaderTest]
at java.base@11/sun.nio.ch.EPoll.wait(Native Method)
at
java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at
java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown
Source)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base@11/java.lang.Thread.run(Thread.java:834)
7) Thread[id=1213, name=coreContainerWorkExecutor-213-thread-1,
state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
at
java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
at
java.base@11/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834)
8) Thread[id=1255, name=Scheduler-1208377519, state=TIMED_WAITING,
group=TGRP-ChaosMonkeySafeLeaderTest]
at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834)
9) Thread[id=923, name=qtp66645842-923, state=RUNNABLE,
group=TGRP-ChaosMonkeySafeLeaderTest]
at java.base@11/sun.nio.ch.EPoll.wait(Native Method)
at
java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at
java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown
Source)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base@11/java.lang.Thread.run(Thread.java:834)
10) Thread[id=1002,
name=qtp1645118601-1002-acceptor-0@cbb0765-ServerConnector@4695d09b{ssl,[ssl,
alpn, http/1.1, h2]}{127.0.0.1:41353}, state=RUNNABLE,
group=TGRP-ChaosMonkeySafeLeaderTest]
at java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept0(Native
Method)
at
java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
at
java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
at
app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385)
at
app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:648)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at
app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base@11/java.lang.Thread.run(Thread.java:834)
11) Thread[id=1229, name=SolrRrdBackendFactory-242-thread-1,
state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
at
java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at
java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
at
java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at
java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at java.base@11/java.util.concurrent.Thread
[...truncated too long message...]
d-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for
registry=solr.cluster, tag=null
[junit4] 2> 130197 INFO (closeThreadPool-306-thread-1) [ ]
o.a.s.c.Overseer Overseer
(id=72167245759119366-127.0.0.1:45179_solr-n_0000000000) closing
[junit4] 2> 130197 INFO
(OverseerStateUpdate-72167245759119366-127.0.0.1:45179_solr-n_0000000000)
[n:127.0.0.1:45179_solr ] o.a.s.c.Overseer Overseer Loop exiting :
127.0.0.1:45179_solr
[junit4] 2> 130197 INFO
(OverseerAutoScalingTriggerThread-72167245759119366-127.0.0.1:45179_solr-n_0000000000)
[ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we
are closed, exiting.
[junit4] 2> 130197 DEBUG (closeThreadPool-306-thread-1) [ ]
o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
[junit4] 2> 130198 DEBUG (closeThreadPool-306-thread-1) [ ]
o.a.s.c.a.ScheduledTriggers Shutting down action executor now
[junit4] 2> 130198 DEBUG (closeThreadPool-306-thread-1) [ ]
o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
[junit4] 2> 130198 DEBUG (closeThreadPool-306-thread-1) [ ]
o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool
executor
[junit4] 2> 130198 DEBUG (closeThreadPool-306-thread-1) [ ]
o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
[junit4] 2> 130198 DEBUG (closeThreadPool-306-thread-1) [ ]
o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 130201 INFO (closeThreadPool-306-thread-2) [ ]
o.a.s.c.Overseer Overseer
(id=72167245759119366-127.0.0.1:45179_solr-n_0000000000) closing
[junit4] 2> 130202 INFO (jetty-closer-253-thread-1) [ ]
o.a.s.c.Overseer Overseer
(id=72167245759119366-127.0.0.1:45179_solr-n_0000000000) closing
[junit4] 2> 130203 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ]
o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid
0x10063ba47e40006, likely client has closed socket
[junit4] 2> 130203 INFO (jetty-closer-253-thread-1) [ ]
o.e.j.s.AbstractConnector Stopped ServerConnector@6e7b5b89{ssl,[ssl, alpn,
http/1.1, h2]}{127.0.0.1:0}
[junit4] 2> 130203 INFO (jetty-closer-253-thread-1) [ ]
o.e.j.s.h.ContextHandler Stopped
o.e.j.s.ServletContextHandler@7694e57c{/solr,null,UNAVAILABLE}
[junit4] 2> 130203 INFO (jetty-closer-253-thread-1) [ ]
o.e.j.s.session node0 Stopped scavenging
[junit4] 2> 130204 INFO
(SUITE-AutoscalingHistoryHandlerTest-seed#[21BB66BA1FDDE245]-worker) [ ]
o.a.s.c.ZkTestServer Shutting down ZkTestServer.
[junit4] 2> 130205 ERROR
(SUITE-AutoscalingHistoryHandlerTest-seed#[21BB66BA1FDDE245]-worker) [ ]
o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper
server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 130205 WARN (ZkTestServer Run Thread) [ ]
o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 6
/solr/collections/AutoscalingHistoryHandlerTest_collection/terms/shard1
[junit4] 2> 6 /solr/aliases.json
[junit4] 2> 3 /solr/collections/.system/terms/shard1
[junit4] 2> 3
/solr/collections/AutoscalingHistoryHandlerTest_collection/state.json
[junit4] 2> 2 /solr/security.json
[junit4] 2> 2
/solr/configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/managed-schema
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 9 /solr/collections/.system/state.json
[junit4] 2> 6 /solr/clusterstate.json
[junit4] 2> 6
/solr/collections/AutoscalingHistoryHandlerTest_collection/state.json
[junit4] 2> 6 /solr/clusterprops.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 6 /solr/live_nodes
[junit4] 2> 6 /solr/collections
[junit4] 2>
[junit4] 2> 130205 INFO
(SUITE-AutoscalingHistoryHandlerTest-seed#[21BB66BA1FDDE245]-worker) [ ]
o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:45849
[junit4] 2> 130205 INFO
(SUITE-AutoscalingHistoryHandlerTest-seed#[21BB66BA1FDDE245]-worker) [ ]
o.a.s.c.ZkTestServer connecting to 127.0.0.1 45849
[junit4] 2> 130205 INFO
(SUITE-AutoscalingHistoryHandlerTest-seed#[21BB66BA1FDDE245]-worker) [ ]
o.a.s.SolrTestCaseJ4 -------------------------------------------------------
Done waiting for tracked resources to be released
[junit4] 2> Dec 10, 2018 2:27:14 AM
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 9 leaked
thread(s).
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene80): {},
docValues:{}, maxPointsInLeafNode=734, maxMBSortInHeap=5.710486773639191,
sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@378836c5),
locale=ta-SG, timezone=SystemV/YST9YDT
[junit4] 2> NOTE: Linux 4.15.0-39-generic amd64/Oracle Corporation 11
(64-bit)/cpus=8,threads=1,free=123447192,total=512229376
[junit4] 2> NOTE: All tests run in this JVM: [ChaosMonkeySafeLeaderTest,
AutoscalingHistoryHandlerTest, AutoscalingHistoryHandlerTest]
[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=AutoscalingHistoryHandlerTest -Dtests.seed=21BB66BA1FDDE245
-Dtests.multiplier=3 -Dtests.slow=true -Dtests.badapples=true
-Dtests.locale=ta-SG -Dtests.timezone=SystemV/YST9YDT -Dtests.asserts=true
-Dtests.file.encoding=UTF-8
[junit4] ERROR 0.00s J2 | AutoscalingHistoryHandlerTest (suite) <<<
[junit4] > Throwable #1: java.lang.RuntimeException: Failed while waiting
for active collection
[junit4] > Timeout waiting to see state for
collection=AutoscalingHistoryHandlerTest_collection :null
[junit4] > Live Nodes: [127.0.0.1:40289_solr, 127.0.0.1:45179_solr]
[junit4] > Last available state: null
[junit4] > at
__randomizedtesting.SeedInfo.seed([21BB66BA1FDDE245]:0)
[junit4] > at
org.apache.solr.cloud.MiniSolrCloudCluster.waitForActiveCollection(MiniSolrCloudCluster.java:736)
[junit4] > at
org.apache.solr.cloud.MiniSolrCloudCluster.waitForActiveCollection(MiniSolrCloudCluster.java:742)
[junit4] > at
org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest.setupCluster(AutoscalingHistoryHandlerTest.java:97)
[junit4] > at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4] > at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[junit4] > at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4] > at
java.base/java.lang.reflect.Method.invoke(Method.java:566)
[junit4] > at java.base/java.lang.Thread.run(Thread.java:834)
[junit4] Completed [9/10 (2!)] on J2 in 45.56s, 0 tests, 1 error <<<
FAILURES!
[junit4]
[junit4] Suite: org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest
[junit4] OK 32.4s J0 | AutoscalingHistoryHandlerTest.testHistory
[junit4] Completed [10/10 (2!)] on J0 in 37.98s, 1 test
[junit4]
[junit4]
[junit4] Tests with failures [seed: 21BB66BA1FDDE245]:
[junit4] - org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest
(suite)
[junit4] - org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest
(suite)
[junit4]
[junit4]
[junit4] JVM J0: 0.71 .. 142.21 = 141.50s
[junit4] JVM J1: 0.71 .. 106.42 = 105.71s
[junit4] JVM J2: 0.70 .. 131.40 = 130.70s
[junit4] Execution time total: 2 minutes 22 seconds
[junit4] Tests summary: 10 suites, 8 tests, 2 suite-level errors
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-http2-Linux/lucene/common-build.xml:1572:
The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-http2-Linux/lucene/common-build.xml:1099:
There were test failures: 10 suites, 8 tests, 2 suite-level errors [seed:
21BB66BA1FDDE245]
Total time: 2 minutes 23 seconds
[repro] Setting last failure code to 256
[repro] Failures:
[repro] 0/5 failed:
org.apache.solr.client.solrj.embedded.SolrExampleStreamingBinaryHttp2Test
[repro] 0/5 failed: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[repro] 2/5 failed:
org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest
[repro] 5/5 failed: org.apache.solr.client.solrj.io.stream.StreamingTest
[repro] Exiting with code 256
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Archiving artifacts
Setting
ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting
ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting
ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all
warnings.
<Git Blamer> GIT_COMMIT=0ecb8110eafb6b33b445f94c0bf171307b5a7a71,
workspace=/var/lib/jenkins/workspace/Lucene-Solr-http2-Linux
[WARNINGS] Computing warning deltas based on reference build #55
Recording test results
Setting
ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting
ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting
ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting
ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting
ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]