[JENKINS] Solr » Solr-Check-cloud2refimpl - Build # 1333 - Still Failing!

Apache Jenkins Server Fri, 12 Mar 2021 13:50:40 -0800

Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-cloud2refimpl/1333/


20 tests failed.
FAILED:  
org.apache.solr.cloud.api.collections.CollectionsAPIDistClusterPerZkTest.classMethod

Error Message:
java.lang.AssertionError: ObjectTracker found 1 object(s) that were not 
released!!! [String]
org.apache.solr.cloud.ZkCollectionTerms@1e781c0c
StackTrace:
null
o.a.s.common.util.ObjectReleaseTracker$ObjectTrackerException: 
o.a.s.cloud.ZkCollectionTerms@1e781c0c
at o.a.s.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:49)
at o.a.s.cloud.ZkCollectionTerms.<init>(ZkCollectionTerms.java:42)
at o.a.s.cloud.ZkController.createCollectionTerms(ZkController.java:1687)
at o.a.s.cloud.ZkController.registerShardTerms(ZkController.java:1657)
at o.a.s.cloud.ZkController.register(ZkController.java:1286)
at o.a.s.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:326)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

 expected null, but was:<ObjectTracker found 1 object(s) that were not 
released!!! [String]
org.apache.solr.cloud.ZkCollectionTerms@1e781c0c
StackTrace:
null
o.a.s.common.util.ObjectReleaseTracker$ObjectTrackerException: 
o.a.s.cloud.ZkCollectionTerms@1e781c0c
at o.a.s.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:49)
at o.a.s.cloud.ZkCollectionTerms.<init>(ZkCollectionTerms.java:42)
at o.a.s.cloud.ZkController.createCollectionTerms(ZkController.java:1687)
at o.a.s.cloud.ZkController.registerShardTerms(ZkController.java:1657)
at o.a.s.cloud.ZkController.register(ZkController.java:1286)
at o.a.s.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:326)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

>

Stack Trace:
java.lang.AssertionError: ObjectTracker found 1 object(s) that were not 
released!!! [String]
org.apache.solr.cloud.ZkCollectionTerms@1e781c0c
StackTrace:
null
o.a.s.common.util.ObjectReleaseTracker$ObjectTrackerException: 
o.a.s.cloud.ZkCollectionTerms@1e781c0c
at o.a.s.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:49)
at o.a.s.cloud.ZkCollectionTerms.<init>(ZkCollectionTerms.java:42)
at o.a.s.cloud.ZkController.createCollectionTerms(ZkController.java:1687)
at o.a.s.cloud.ZkController.registerShardTerms(ZkController.java:1657)
at o.a.s.cloud.ZkController.register(ZkController.java:1286)
at o.a.s.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:326)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

 expected null, but was:<ObjectTracker found 1 object(s) that were not 
released!!! [String]
org.apache.solr.cloud.ZkCollectionTerms@1e781c0c
StackTrace:
null
o.a.s.common.util.ObjectReleaseTracker$ObjectTrackerException: 
o.a.s.cloud.ZkCollectionTerms@1e781c0c
at o.a.s.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:49)
at o.a.s.cloud.ZkCollectionTerms.<init>(ZkCollectionTerms.java:42)
at o.a.s.cloud.ZkController.createCollectionTerms(ZkController.java:1687)
at o.a.s.cloud.ZkController.registerShardTerms(ZkController.java:1657)
at o.a.s.cloud.ZkController.register(ZkController.java:1286)
at o.a.s.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:326)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

>
        at __randomizedtesting.SeedInfo.seed([506D1D5A107A4206]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotNull(Assert.java:756)
        at org.junit.Assert.assertNull(Assert.java:738)
        at 
org.apache.solr.SolrTestCase.afterSolrTestCaseClass(SolrTestCase.java:572)
        at jdk.internal.reflect.GeneratedMethodAccessor51.invoke(Unknown Source)
        at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:905)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.solr.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:406)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:712)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:138)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:633)


FAILED:  
org.apache.solr.cloud.api.collections.CollectionsAPIDistClusterPerZkTest.classMethod

Error Message:
com.carrotsearch.randomizedtesting.ThreadLeakError: 175 threads leaked from 
SUITE scope at 
org.apache.solr.cloud.api.collections.CollectionsAPIDistClusterPerZkTest: 
   1) Thread[id=5801, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5175, 
state=TERMINATED, group={null group}]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
   2) Thread[id=5958, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5315, 
state=TERMINATED, group={null group}]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
   3) Thread[id=5862, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5228, 
state=TERMINATED, group={null group}]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
   4) Thread[id=5809, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5183, 
state=TERMINATED, group={null group}]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
   5) Thread[id=5944, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5301, 
state=TERMINATED, group={null group}]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
   6) Thread[id=5731, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5123, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistClusterPerZkTest]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
   7) Thread[id=5737, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5129, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistClusterPerZkTest]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
   8) Thread[id=5938, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5295, 
state=TERMINATED, group={null group}]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
   9) Thread[id=5943, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5300, 
state=TERMINATED, group={null group}]
        at (empty stack)
  10) Thread[id=5774, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5148, 
state=TERMINATED, group={null group}]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  11) Thread[id=5841, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5210, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistClusterPerZkTest]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  12) Thread[id=5893, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5257, 
state=TERMINATED, group={null group}]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  13) Thread[id=5770, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5145, 
state=TERMINATED, group={null group}]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  14) Thread[id=5821, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5195, 
state=TERMINATED, group={null group}]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  15) Thread[id=5927, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5284, 
state=TERMINATED, group={null group}]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  16) Thread[id=5932, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5289, 
state=TERMINATED, group={null group}]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  17) Thread[id=5798, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5172, 
state=TERMINATED, group={null group}]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  18) Thread[id=5849, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5217, 
state=RUNNABLE, group=TGRP-CollectionsAPIDistClusterPerZkTest]
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:463)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  19) Thread[id=5931, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5288, 
state=TERMINATED, group={null group}]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  20) Thread[id=5914, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5271, 
state=TERMINATED, group={null group}]
        at (empty stack)
  21) Thread[id=5860, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5226, 
state=TERMINATED, group={null group}]
        at (empty stack)
  22) Thread[id=5901, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5265, 
state=TERMINATED, group={null group}]
        at (empty stack)
  23) Thread[id=5948, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5305, 
state=TERMINATED, group={null group}]
        at (empty stack)
  24) Thread[id=5909, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5268, 
state=TERMINATED, group={null group}]
        at (empty stack)
  25) Thread[id=5885, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5249, 
state=TERMINATED, group={null group}]
        at (empty stack)
  26) Thread[id=5939, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5296, 
state=TERMINATED, group={null group}]
        at (empty stack)
  27) Thread[id=5859, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5225, 
state=TERMINATED, group={null group}]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  28) Thread[id=5816, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5190, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistClusterPerZkTest]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  29) Thread[id=5888, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5252, 
state=TERMINATED, group={null group}]
        at (empty stack)
  30) Thread[id=5899, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5263, 
state=TERMINATED, group={null group}]
        at (empty stack)
  31) Thread[id=5772, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5146, 
state=TERMINATED, group={null group}]
        at (empty stack)
  32) Thread[id=5733, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5125, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistClusterPerZkTest]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  33) Thread[id=5908, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5267, 
state=TERMINATED, group={null group}]
        at (empty stack)
  34) Thread[id=5917, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5274, 
state=TERMINATED, group={null group}]
        at (empty stack)
  35) Thread[id=5856, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5223, 
state=TERMINATED, group={null group}]
        at (empty stack)
  36) Thread[id=5880, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5246, 
state=TERMINATED, group={null group}]
        at (empty stack)
  37) Thread[id=5802, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5176, 
state=TERMINATED, group={null group}]
        at (empty stack)
  38) Thread[id=5854, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5221, 
state=TERMINATED, group={null group}]
        at (empty stack)
  39) Thread[id=5790, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5164, 
state=TERMINATED, group={null group}]
        at (empty stack)
  40) Thread[id=5796, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5170, 
state=TERMINATED, group={null group}]
        at (empty stack)
  41) Thread[id=5740, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5132, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistClusterPerZkTest]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  42) Thread[id=5815, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5189, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistClusterPerZkTest]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  43) Thread[id=5730, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5122, 
state=TERMINATED, group={null group}]
        at (empty stack)
  44) Thread[id=5889, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5253, 
state=TERMINATED, group={null group}]
        at (empty stack)
  45) Thread[id=5805, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5179, 
state=TERMINATED, group={null group}]
        at (empty stack)
  46) Thread[id=5807, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5181, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistClusterPerZkTest]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  47) Thread[id=5892, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5256, 
state=TERMINATED, group={null group}]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  48) Thread[id=5895, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5259, 
state=TERMINATED, group={null group}]
        at (empty stack)
  49) Thread[id=5875, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5241, 
state=TERMINATED, group={null group}]
        at (empty stack)
  50) Thread[id=5787, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5161, 
state=TERMINATED, group={null group}]
        at (empty stack)
  51) Thread[id=5915, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5272, 
state=TERMINATED, group={null group}]
        at (empty stack)
  52) Thread[id=5876, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5243, 
state=TERMINATED, group={null group}]
        at (empty stack)
  53) Thread[id=5919, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5276, 
state=TERMINATED, group={null group}]
        at (empty stack)
  54) Thread[id=5735, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5127, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistClusterPerZkTest]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  55) Thread[id=5813, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5187, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistClusterPerZkTest]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  56) Thread[id=5910, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5269, 
state=TERMINATED, group={null group}]
        at (empty stack)
  57) Thread[id=5810, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5184, 
state=TERMINATED, group={null group}]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  58) Thread[id=5846, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5214, 
state=TERMINATED, group={null group}]
        at (empty stack)
  59) Thread[id=5881, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5242, 
state=TERMINATED, group={null group}]
        at (empty stack)
  60) Thread[id=5928, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5286, 
state=TERMINATED, group={null group}]
        at (empty stack)
  61) Thread[id=5920, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5277, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistClusterPerZkTest]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  62) Thread[id=5874, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5240, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistClusterPerZkTest]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  63) Thread[id=5890, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5254, 
state=TERMINATED, group={null group}]
        at (empty stack)
  64) Thread[id=5945, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5302, 
state=TERMINATED, group={null group}]
        at (empty stack)
  65) Thread[id=5822, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5196, 
state=TERMINATED, group={null group}]
        at (empty stack)
  66) Thread[id=5786, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5160, 
state=TERMINATED, group={null group}]
        at (empty stack)
  67) Thread[id=5865, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5231, 
state=TERMINATED, group={null group}]
        at (empty stack)
  68) Thread[id=5791, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5165, 
state=TERMINATED, group={null group}]
        at (empty stack)
  69) Thread[id=5795, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5169, 
state=TERMINATED, group={null group}]
        at (empty stack)
  70) Thread[id=5898, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5262, 
state=TERMINATED, group={null group}]
        at (empty stack)
  71) Thread[id=5877, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5244, 
state=TERMINATED, group={null group}]
        at (empty stack)
  72) Thread[id=5882, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5247, 
state=TERMINATED, group={null group}]
        at (empty stack)
  73) Thread[id=5935, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5292, 
state=TERMINATED, group={null group}]
        at (empty stack)
  74) Thread[id=5947, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5304, 
state=TERMINATED, group={null group}]
        at (empty stack)
  75) Thread[id=5817, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5191, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistClusterPerZkTest]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  76) Thread[id=5803, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5177, 
state=TERMINATED, group={null group}]
        at (empty stack)
  77) Thread[id=5891, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5255, 
state=TERMINATED, group={null group}]
        at (empty stack)
  78) Thread[id=5845, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5213, 
state=TERMINATED, group={null group}]
        at (empty stack)
  79) Thread[id=5907, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5266, 
state=TERMINATED, group={null group}]
        at (empty stack)
  80) Thread[id=5921, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5278, 
state=TERMINATED, group={null group}]
        at (empty stack)
  81) Thread[id=5949, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5306, 
state=TERMINATED, group={null group}]
        at (empty stack)
  82) Thread[id=5896, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5260, 
state=TERMINATED, group={null group}]
        at (empty stack)
  83) Thread[id=5952, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5309, 
state=TERMINATED, group={null group}]
        at (empty stack)
  84) Thread[id=5850, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5218, 
state=TERMINATED, group={null group}]
        at (empty stack)
  85) Thread[id=5768, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5142, 
state=TERMINATED, group={null group}]
        at (empty stack)
  86) Thread[id=5861, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5227, 
state=TERMINATED, group={null group}]
        at (empty stack)
  87) Thread[id=5781, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5155, 
state=TERMINATED, group={null group}]
        at (empty stack)
  88) Thread[id=5884, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5248, 
state=TERMINATED, group={null group}]
        at (empty stack)
  89) Thread[id=5800, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5174, 
state=TERMINATED, group={null group}]
        at (empty stack)
  90) Thread[id=5957, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5313, 
state=TERMINATED, group={null group}]
        at (empty stack)
  91) Thread[id=5738, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5130, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistClusterPerZkTest]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  92) Thread[id=5869, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5235, 
state=TERMINATED, group={null group}]
        at (empty stack)
  93) Thread[id=5879, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5245, 
state=TERMINATED, group={null group}]
        at (empty stack)
  94) Thread[id=5924, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5281, 
state=TERMINATED, group={null group}]
        at (empty stack)
  95) Thread[id=5926, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5283, 
state=TERMINATED, group={null group}]
        at (empty stack)
  96) Thread[id=5923, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5280, 
state=TERMINATED, group={null group}]
        at (empty stack)
  97) Thread[id=5853, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5220, 
state=TERMINATED, group={null group}]
        at (empty stack)
  98) Thread[id=5887, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5251, 
state=TERMINATED, group={null group}]
        at (empty stack)
  99) Thread[id=5900, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5264, 
state=TERMINATED, group={null group}]
        at (empty stack)
  100) Thread[id=5785, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5159, 
state=TERMINATED, group={null group}]
        at (empty stack)
  101) Thread[id=5941, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5298, 
state=TERMINATED, group={null group}]
        at (empty stack)
  102) Thread[id=5942, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5299, 
state=TERMINATED, group={null group}]
        at (empty stack)
  103) Thread[id=5858, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5224, 
state=TERMINATED, group={null group}]
        at (empty stack)
  104) Thread[id=5852, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5219, 
state=TERMINATED, group={null group}]
        at (empty stack)
  105) Thread[id=5940, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5297, 
state=TERMINATED, group={null group}]
        at (empty stack)
  106) Thread[id=5780, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5154, 
state=TERMINATED, group={null group}]
        at (empty stack)
  107) Thread[id=5913, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5270, 
state=TERMINATED, group={null group}]
        at (empty stack)
  108) Thread[id=5736, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5128, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistClusterPerZkTest]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  109) Thread[id=5863, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5229, 
state=TERMINATED, group={null group}]
        at (empty stack)
  110) Thread[id=5955, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5312, 
state=TERMINATED, group={null group}]
        at (empty stack)
  111) Thread[id=5782, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5156, 
state=TERMINATED, group={null group}]
        at (empty stack)
  112) Thread[id=5837, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5207, 
state=TERMINATED, group={null group}]
        at (empty stack)
  113) Thread[id=5793, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5167, 
state=TERMINATED, group={null group}]
        at (empty stack)
  114) Thread[id=5953, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5310, 
state=TERMINATED, group={null group}]
        at (empty stack)
  115) Thread[id=5937, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5294, 
state=TERMINATED, group={null group}]
        at (empty stack)
  116) Thread[id=5808, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5182, 
state=TERMINATED, group={null group}]
        at (empty stack)
  117) Thread[id=5929, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5285, 
state=TERMINATED, group={null group}]
        at (empty stack)
  118) Thread[id=5794, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5168, 
state=TERMINATED, group={null group}]
        at (empty stack)
  119) Thread[id=5788, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5162, 
state=TERMINATED, group={null group}]
        at (empty stack)
  120) Thread[id=5792, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5166, 
state=TERMINATED, group={null group}]
        at (empty stack)
  121) Thread[id=5894, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5258, 
state=TERMINATED, group={null group}]
        at (empty stack)
  122) Thread[id=5925, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5282, 
state=TERMINATED, group={null group}]
        at (empty stack)
  123) Thread[id=5930, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5287, 
state=TERMINATED, group={null group}]
        at (empty stack)
  124) Thread[id=5934, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5291, 
state=TERMINATED, group={null group}]
        at (empty stack)
  125) Thread[id=5824, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5198, 
state=TERMINATED, group={null group}]
        at (empty stack)
  126) Thread[id=5864, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5230, 
state=TERMINATED, group={null group}]
        at (empty stack)
  127) Thread[id=5806, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5180, 
state=TERMINATED, group={null group}]
        at (empty stack)
  128) Thread[id=5823, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5197, 
state=TERMINATED, group={null group}]
        at (empty stack)
  129) Thread[id=5855, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5222, 
state=TERMINATED, group={null group}]
        at (empty stack)
  130) Thread[id=5844, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5212, 
state=TERMINATED, group={null group}]
        at (empty stack)
  131) Thread[id=5868, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5234, 
state=TERMINATED, group={null group}]
        at (empty stack)
  132) Thread[id=5734, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5126, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistClusterPerZkTest]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  133) Thread[id=5871, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5237, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistClusterPerZkTest]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  134) Thread[id=5922, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5279, 
state=TERMINATED, group={null group}]
        at (empty stack)
  135) Thread[id=5814, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5188, 
state=TERMINATED, group={null group}]
        at (empty stack)
  136) Thread[id=5933, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5290, 
state=TERMINATED, group={null group}]
        at (empty stack)
  137) Thread[id=5954, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5311, 
state=TERMINATED, group={null group}]
        at (empty stack)
  138) Thread[id=5799, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5173, 
state=TERMINATED, group={null group}]
        at (empty stack)
  139) Thread[id=5819, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5193, 
state=TERMINATED, group={null group}]
        at (empty stack)
  140) Thread[id=5951, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5308, 
state=TERMINATED, group={null group}]
        at (empty stack)
  141) Thread[id=5839, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5208, 
state=TERMINATED, group={null group}]
        at (empty stack)
  142) Thread[id=5739, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5131, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistClusterPerZkTest]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  143) Thread[id=5959, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5316, 
state=TERMINATED, group={null group}]
        at (empty stack)
  144) Thread[id=5842, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5211, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistClusterPerZkTest]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  145) Thread[id=5866, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5232, 
state=TERMINATED, group={null group}]
        at (empty stack)
  146) Thread[id=5956, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5314, 
state=TERMINATED, group={null group}]
        at (empty stack)
  147) Thread[id=5886, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5250, 
state=TERMINATED, group={null group}]
        at (empty stack)
  148) Thread[id=5811, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5185, 
state=TERMINATED, group={null group}]
        at (empty stack)
  149) Thread[id=5847, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5215, 
state=TERMINATED, group={null group}]
        at (empty stack)
  150) Thread[id=5870, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5236, 
state=TERMINATED, group={null group}]
        at (empty stack)
  151) Thread[id=5804, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5178, 
state=TERMINATED, group={null group}]
        at (empty stack)
  152) Thread[id=5773, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5147, 
state=TERMINATED, group={null group}]
        at (empty stack)
  153) Thread[id=5916, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5273, 
state=TERMINATED, group={null group}]
        at (empty stack)
  154) Thread[id=5732, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5124, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistClusterPerZkTest]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  155) Thread[id=5897, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5261, 
state=TERMINATED, group={null group}]
        at (empty stack)
  156) Thread[id=5872, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5238, 
state=TERMINATED, group={null group}]
        at (empty stack)
  157) Thread[id=5867, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5233, 
state=TERMINATED, group={null group}]
        at (empty stack)
  158) Thread[id=5789, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5163, 
state=TERMINATED, group={null group}]
        at (empty stack)
  159) Thread[id=5812, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5186, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistClusterPerZkTest]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  160) Thread[id=5873, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5239, 
state=TERMINATED, group={null group}]
        at (empty stack)
  161) Thread[id=5918, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5275, 
state=TERMINATED, group={null group}]
        at (empty stack)
  162) Thread[id=5840, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5209, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistClusterPerZkTest]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  163) Thread[id=5820, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5194, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistClusterPerZkTest]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  164) Thread[id=5797, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5171, 
state=TERMINATED, group={null group}]
        at (empty stack)
  165) Thread[id=5818, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5192, 
state=TERMINATED, group={null group}]
        at (empty stack)
  166) Thread[id=5848, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5216, 
state=TERMINATED, group={null group}]
        at (empty stack)
  167) Thread[id=5946, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5303, 
state=TERMINATED, group={null group}]
        at (empty stack)
  168) Thread[id=5950, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5307, 
state=TERMINATED, group={null group}]
        at (empty stack)
  169) Thread[id=5783, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5157, 
state=TERMINATED, group={null group}]
        at (empty stack)
  170) Thread[id=5784, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5158, 
state=TERMINATED, group={null group}]
        at (empty stack)
  171) Thread[id=5729, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5121, 
state=TERMINATED, group={null group}]
        at (empty stack)
  172) Thread[id=5936, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5293, 
state=TERMINATED, group={null group}]
        at (empty stack)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 175 threads leaked from 
SUITE scope at 
org.apache.solr.cloud.api.collections.CollectionsAPIDistClusterPerZkTest: 
   1) Thread[id=5801, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5175, 
state=TERMINATED, group={null group}]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
   2) Thread[id=5958, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5315, 
state=TERMINATED, group={null group}]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
   3) Thread[id=5862, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5228, 
state=TERMINATED, group={null group}]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
   4) Thread[id=5809, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5183, 
state=TERMINATED, group={null group}]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
   5) Thread[id=5944, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5301, 
state=TERMINATED, group={null group}]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
   6) Thread[id=5731, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5123, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistClusterPerZkTest]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
   7) Thread[id=5737, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5129, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistClusterPerZkTest]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
   8) Thread[id=5938, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5295, 
state=TERMINATED, group={null group}]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
   9) Thread[id=5943, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5300, 
state=TERMINATED, group={null group}]
        at (empty stack)
  10) Thread[id=5774, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5148, 
state=TERMINATED, group={null group}]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  11) Thread[id=5841, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5210, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistClusterPerZkTest]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  12) Thread[id=5893, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5257, 
state=TERMINATED, group={null group}]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  13) Thread[id=5770, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5145, 
state=TERMINATED, group={null group}]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  14) Thread[id=5821, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5195, 
state=TERMINATED, group={null group}]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  15) Thread[id=5927, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5284, 
state=TERMINATED, group={null group}]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  16) Thread[id=5932, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5289, 
state=TERMINATED, group={null group}]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  17) Thread[id=5798, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5172, 
state=TERMINATED, group={null group}]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  18) Thread[id=5849, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5217, 
state=RUNNABLE, group=TGRP-CollectionsAPIDistClusterPerZkTest]
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:463)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  19) Thread[id=5931, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5288, 
state=TERMINATED, group={null group}]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  20) Thread[id=5914, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5271, 
state=TERMINATED, group={null group}]
        at (empty stack)
  21) Thread[id=5860, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5226, 
state=TERMINATED, group={null group}]
        at (empty stack)
  22) Thread[id=5901, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5265, 
state=TERMINATED, group={null group}]
        at (empty stack)
  23) Thread[id=5948, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5305, 
state=TERMINATED, group={null group}]
        at (empty stack)
  24) Thread[id=5909, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5268, 
state=TERMINATED, group={null group}]
        at (empty stack)
  25) Thread[id=5885, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5249, 
state=TERMINATED, group={null group}]
        at (empty stack)
  26) Thread[id=5939, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5296, 
state=TERMINATED, group={null group}]
        at (empty stack)
  27) Thread[id=5859, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5225, 
state=TERMINATED, group={null group}]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at 
[email protected]/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at 
app//org.apache.solr.common.ParWorkExecutor$ParWorkThreadFactory$1.run(ParWorkExecutor.java:117)
  28) Thread[id=5816, name=SOLR-TGRP-CollectionsAPIDistClusterPerZkTest-5190, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistClusterPerZkTest]
        at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
        at 
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at 
[email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at [email protected]/java.util.concurrent.Synchr

[...truncated too long message...]

  Found an unexpected log level: istraceenabled, file: 
/home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-cloud2refimpl/solr/solrj/src/java/org/apache/solr/common/ParWork.java:468,
 
  cause: 'hasParens true prevLineNotIf true pernicketyLevel true' Suspicious 
logging call, Parameterize and possibly surround with 'if (log.is*Enabled) 
{..}'. Help at: 'gradlew helpValidateLogCalls' 
      
/home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-cloud2refimpl/solr/solrj/src/java/org/apache/solr/common/ParWork.java:468,
 
  cause: 'hasPlus: true' Suspicious logging call, Parameterize and possibly 
surround with 'if (log.is*Enabled) {..}'. Help at: 'gradlew 
helpValidateLogCalls' 
      
/home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-cloud2refimpl/solr/solrj/src/java/org/apache/solr/common/ParWork.java:511,
 
  cause: 'hasPlus: true' Suspicious logging call, Parameterize and possibly 
surround with 'if (log.is*Enabled) {..}'. Help at: 'gradlew 
helpValidateLogCalls' 
      
/home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-cloud2refimpl/solr/solrj/src/java/org/apache/solr/common/ParWork.java:516,
 
  Found an unexpected log level: istraceenabled, file: 
/home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-cloud2refimpl/solr/solrj/src/java/org/apache/solr/common/ParWork.java:532,
 
  cause: 'hasParens true prevLineNotIf true pernicketyLevel true' Suspicious 
logging call, Parameterize and possibly surround with 'if (log.is*Enabled) 
{..}'. Help at: 'gradlew helpValidateLogCalls' 
      
/home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-cloud2refimpl/solr/solrj/src/java/org/apache/solr/common/ParWork.java:532,
 
  cause: 'hasPlus: true' Suspicious logging call, Parameterize and possibly 
surround with 'if (log.is*Enabled) {..}'. Help at: 'gradlew 
helpValidateLogCalls' 
      
/home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-cloud2refimpl/solr/solrj/src/java/org/apache/solr/common/ParWork.java:562,
 
  cause: 'hasPlus: true' Suspicious logging call, Parameterize and possibly 
surround with 'if (log.is*Enabled) {..}'. Help at: 'gradlew 
helpValidateLogCalls' 
      
/home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-cloud2refimpl/solr/solrj/src/java/org/apache/solr/common/ParWork.java:564,
 
  Change the logger name to lower-case 'log' in SolrQueuedThreadPool.java   
private static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass()); projectproject 
':solr:solrj', 
  Change the logger name to lower-case 'log' in ZooKeeperExposed.java     
private static final Logger LOG = 
LoggerFactory.getLogger(ZooKeeperTestable.class); projectproject ':solr:solrj', 
  cause: 'hasParens true prevLineNotIf true pernicketyLevel true' Suspicious 
logging call, Parameterize and possibly surround with 'if (log.is*Enabled) 
{..}'. Help at: 'gradlew helpValidateLogCalls' 
      
/home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-cloud2refimpl/solr/solrj/src/test/org/apache/solr/client/solrj/impl/BaseSolrClientWireMockTest.java:156,
 
  Change the logger name to lower-case 'log' in SolrProvider.java   private 
static final Logger LOGGER = StatusLogger.getLogger(); projectproject 
':solr:solrj').

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.
==============================================================================

40: Task failed with an exception.
-----------
* Where:
Script 
'/home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-cloud2refimpl/gradle/validation/validate-source-patterns.gradle'
 line: 324

* What went wrong:
Execution failed for task ':solr:solrj:validateSourcePatterns'.
> Found 20 violations in source files (invalid logger name [log, uses static 
> class name, not specialized logger], invalid logging pattern [not private 
> static final, uses static class name]).

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.
==============================================================================

41: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:test-framework:ecjLintMain'.
> Process 'command '/usr/local/asfpackages/java/oraclejdk-11.0.9/bin/java'' 
> finished with non-zero exit value 255

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.
==============================================================================

42: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:test-framework:forbiddenApisMain'.
> de.thetaphi.forbiddenapis.ForbiddenApiException: Check for forbidden API 
> calls failed, see log.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.
==============================================================================

43: Task failed with an exception.
-----------
* Where:
Script 
'/home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-cloud2refimpl/gradle/validation/validate-log-calls.gradle'
 line: 243

* What went wrong:
Execution failed for task ':solr:test-framework:validateLogCalls'.
> Found 1 violations in source files (
  cause: 'hasParens true prevLineNotIf true pernicketyLevel true' Suspicious 
logging call, Parameterize and possibly surround with 'if (log.is*Enabled) 
{..}'. Help at: 'gradlew helpValidateLogCalls' 
      
/home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-cloud2refimpl/solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java:296).

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.
==============================================================================

44: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:contrib:analytics:ecjLintTest'.
> Process 'command '/usr/local/asfpackages/java/oraclejdk-11.0.9/bin/java'' 
> finished with non-zero exit value 255

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.
==============================================================================

45: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:contrib:dataimporthandler:ecjLintMain'.
> Process 'command '/usr/local/asfpackages/java/oraclejdk-11.0.9/bin/java'' 
> finished with non-zero exit value 255

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.
==============================================================================

46: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:contrib:dataimporthandler:ecjLintTest'.
> Process 'command '/usr/local/asfpackages/java/oraclejdk-11.0.9/bin/java'' 
> finished with non-zero exit value 255

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.
==============================================================================

47: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:contrib:dataimporthandler-extras:ecjLintMain'.
> Process 'command '/usr/local/asfpackages/java/oraclejdk-11.0.9/bin/java'' 
> finished with non-zero exit value 255

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.
==============================================================================

48: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:contrib:extraction:ecjLintMain'.
> Process 'command '/usr/local/asfpackages/java/oraclejdk-11.0.9/bin/java'' 
> finished with non-zero exit value 255

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.
==============================================================================

49: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:contrib:extraction:ecjLintTest'.
> Process 'command '/usr/local/asfpackages/java/oraclejdk-11.0.9/bin/java'' 
> finished with non-zero exit value 255

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.
==============================================================================

50: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:contrib:ltr:ecjLintTest'.
> Process 'command '/usr/local/asfpackages/java/oraclejdk-11.0.9/bin/java'' 
> finished with non-zero exit value 255

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with 
Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See 
https://docs.gradle.org/6.4.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 20m 18s
788 actionable tasks: 788 executed
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-cloud2refimpl - Build # 1333 - Still Failing!

Reply via email to