[JENKINS] Solr » Solr-Check-9.x - Build # 5411 - Still Failing!

[Apache Jenkins Server]

Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/5411/

3 tests failed.
FAILED:  org.apache.solr.cloud.MultiThreadedOCPTest.classMethod

Error Message:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE 
scope at org.apache.solr.cloud.MultiThreadedOCPTest: 
   1) Thread[id=559, name=aliveCheckExecutor-336-thread-1, state=TIMED_WAITING, 
group=TGRP-MultiThreadedOCPTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at 
java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE 
scope at org.apache.solr.cloud.MultiThreadedOCPTest: 
   1) Thread[id=559, name=aliveCheckExecutor-336-thread-1, state=TIMED_WAITING, 
group=TGRP-MultiThreadedOCPTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at 
java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
        at __randomizedtesting.SeedInfo.seed([FBE234975B28C38F]:0)


FAILED:  org.apache.solr.cloud.MultiThreadedOCPTest.classMethod

Error Message:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie 
threads that couldn't be terminated:
   1) Thread[id=559, name=aliveCheckExecutor-336-thread-1, state=TIMED_WAITING, 
group=TGRP-MultiThreadedOCPTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at 
java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie 
threads that couldn't be terminated:
   1) Thread[id=559, name=aliveCheckExecutor-336-thread-1, state=TIMED_WAITING, 
group=TGRP-MultiThreadedOCPTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at 
java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at 
java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
        at __randomizedtesting.SeedInfo.seed([FBE234975B28C38F]:0)


FAILED:  
org.apache.solr.client.solrj.request.json.DirectJsonQueryRequestFacetingIntegrationTest.classMethod

Error Message:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([FBE234975B28C38F]:0)




Build Log:
[...truncated 7294 lines...]
# ERROR: Solr is still running at http://localhost:7574/solr after 5 seconds
#
# Please find the SOLR_HOME snapshot for failed test #2 at: 
/tmp/src/solr/solr/packaging/build/test-output/failure-snapshots/62-2
# Last output:
#
# Found 2 Solr nodes running! Must either specify a port using -p or -all to 
stop all Solr nodes on this host.
ok 63 status detects locally running solr in 14355ms
ok 64 status does not expose cli parameters to end user in 207ms
ok 65 listing out files in 2156ms
ok 66 copying files around in 6527ms
ok 67 upconfig in 2625ms
ok 68 zkcli.sh gets 'solrhome' from 'solr.home' system property in 1221ms
ok 69 zkcli.sh gets 'solrhome' from 'solrhome' command line option in 1222ms

> Task :solr:packaging:integrationTests FAILED

> Task :solr:solrj:test

org.apache.solr.client.solrj.request.json.DirectJsonQueryRequestFacetingIntegrationTest
 > classMethod FAILED
    java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([FBE234975B28C38F]:0)

org.apache.solr.client.solrj.request.json.DirectJsonQueryRequestFacetingIntegrationTest
 > test suite's output saved to 
/tmp/src/solr/solr/solrj/build/test-results/test/outputs/OUTPUT-org.apache.solr.client.solrj.request.json.DirectJsonQueryRequestFacetingIntegrationTest.txt,
 copied below:
  2> 2829 INFO  
(SUITE-DirectJsonQueryRequestFacetingIntegrationTest-seed#[FBE234975B28C38F]-worker)
 [] o.a.s.SolrTestCase Setting 'solr.default.confdir' system property to 
test-framework derived value of 
'/tmp/src/solr/solr/server/solr/configsets/_default/conf'
  2> 2842 INFO  
(SUITE-DirectJsonQueryRequestFacetingIntegrationTest-seed#[FBE234975B28C38F]-worker)
 [] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: 
test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
  2> 2873 INFO  
(SUITE-DirectJsonQueryRequestFacetingIntegrationTest-seed#[FBE234975B28C38F]-worker)
 [] o.a.s.u.ErrorLogMuter Creating ErrorLogMuter-regex-1 for ERROR logs 
matching regex: ignore_exception
  2> 2899 INFO  
(SUITE-DirectJsonQueryRequestFacetingIntegrationTest-seed#[FBE234975B28C38F]-worker)
 [] o.a.s.SolrTestCaseJ4 Created dataDir: 
/tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.request.json.DirectJsonQueryRequestFacetingIntegrationTest_FBE234975B28C38F-001/data-dir-1-001
  2> 3001 INFO  
(SUITE-DirectJsonQueryRequestFacetingIntegrationTest-seed#[FBE234975B28C38F]-worker)
 [] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) 
w/NUMERIC_DOCVALUES_SYSPROP=true
  2> 3047 INFO  
(SUITE-DirectJsonQueryRequestFacetingIntegrationTest-seed#[FBE234975B28C38F]-worker)
 [] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: 
@org.apache.solr.util.RandomizeSSL(reason="", value=0.0/0.0, ssl=0.0/0.0, 
clientAuth=0.0/0.0)
  2> 3093 INFO  
(SUITE-DirectJsonQueryRequestFacetingIntegrationTest-seed#[FBE234975B28C38F]-worker)
 [] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
  2> 3095 INFO  
(SUITE-DirectJsonQueryRequestFacetingIntegrationTest-seed#[FBE234975B28C38F]-worker)
 [] o.a.s.c.MiniSolrCloudCluster Starting cluster of 1 servers in 
/tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.request.json.DirectJsonQueryRequestFacetingIntegrationTest_FBE234975B28C38F-001/tempDir-001
  2> 3099 WARN  
(SUITE-DirectJsonQueryRequestFacetingIntegrationTest-seed#[FBE234975B28C38F]-worker)
 [] o.a.s.c.ZkTestServer TEST_PATH() does not exist - solrj test?
  2> 3104 INFO  
(SUITE-DirectJsonQueryRequestFacetingIntegrationTest-seed#[FBE234975B28C38F]-worker)
 [] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
  2> 3113 INFO  (ZkTestServer Run Thread) [] o.a.s.c.ZkTestServer client port: 
0.0.0.0/0.0.0.0:0
  2> 3114 INFO  (ZkTestServer Run Thread) [] o.a.s.c.ZkTestServer Starting 
server
  2> 3363 WARN  (ZkTestServer Run Thread) [] o.a.z.s.ServerCnxnFactory maxCnxns 
is not configured, using default value 0.
  2> 3429 INFO  
(SUITE-DirectJsonQueryRequestFacetingIntegrationTest-seed#[FBE234975B28C38F]-worker)
 [] o.a.s.c.ZkTestServer start zk server on port: 40721
  2> 3795 WARN  
(SUITE-DirectJsonQueryRequestFacetingIntegrationTest-seed#[FBE234975B28C38F]-worker)
 [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. 
ZkCredentialsInjector is not secure, it creates an empty list of credentials 
which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 3904 INFO  
(SUITE-DirectJsonQueryRequestFacetingIntegrationTest-seed#[FBE234975B28C38F]-worker)
 [] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to 
ZooKeeper
  2> 4001 INFO  (zkConnectionManagerCallback-3-thread-1) [] 
o.a.s.c.c.ConnectionManager zkClient has connected
  2> 4002 INFO  
(SUITE-DirectJsonQueryRequestFacetingIntegrationTest-seed#[FBE234975B28C38F]-worker)
 [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 4002 WARN  
(SUITE-DirectJsonQueryRequestFacetingIntegrationTest-seed#[FBE234975B28C38F]-worker)
 [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is 
not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 4035 WARN  
(SUITE-DirectJsonQueryRequestFacetingIntegrationTest-seed#[FBE234975B28C38F]-worker)
 [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. 
ZkCredentialsInjector is not secure, it creates an empty list of credentials 
which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 4046 INFO  
(SUITE-DirectJsonQueryRequestFacetingIntegrationTest-seed#[FBE234975B28C38F]-worker)
 [] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to 
ZooKeeper
  2> 4090 INFO  (zkConnectionManagerCallback-5-thread-1) [] 
o.a.s.c.c.ConnectionManager zkClient has connected
  2> 4091 INFO  
(SUITE-DirectJsonQueryRequestFacetingIntegrationTest-seed#[FBE234975B28C38F]-worker)
 [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 4091 WARN  
(SUITE-DirectJsonQueryRequestFacetingIntegrationTest-seed#[FBE234975B28C38F]-worker)
 [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is 
not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 4092 WARN  
(SUITE-DirectJsonQueryRequestFacetingIntegrationTest-seed#[FBE234975B28C38F]-worker)
 [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. 
ZkCredentialsInjector is not secure, it creates an empty list of credentials 
which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 4114 INFO  
(SUITE-DirectJsonQueryRequestFacetingIntegrationTest-seed#[FBE234975B28C38F]-worker)
 [] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to 
ZooKeeper
  2> 4135 INFO  (zkConnectionManagerCallback-7-thread-1) [] 
o.a.s.c.c.ConnectionManager zkClient has connected
  2> 4136 INFO  
(SUITE-DirectJsonQueryRequestFacetingIntegrationTest-seed#[FBE234975B28C38F]-worker)
 [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 4136 WARN  
(SUITE-DirectJsonQueryRequestFacetingIntegrationTest-seed#[FBE234975B28C38F]-worker)
 [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is 
not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 4671 INFO  (jetty-launcher-8-thread-1) [] o.a.s.e.JettySolrRunner Start 
Jetty (configured port=0, binding port=0)
  2> 4675 INFO  (jetty-launcher-8-thread-1) [] o.a.s.e.JettySolrRunner Trying 
to start Jetty on port 0 try number 2 ...
  2> 4696 INFO  (jetty-launcher-8-thread-1) [] o.e.j.s.Server jetty-10.0.16; 
built: 2023-08-25T19:11:25.422Z; git: a2735a9ae9d1afc124974253f1e223e5678be1f4; 
jvm 11.0.16+8
  2> 4849 INFO  (jetty-launcher-8-thread-1) [] 
o.e.j.s.s.DefaultSessionIdManager Session workerName=node0
  2> 4882 INFO  (jetty-launcher-8-thread-1) [] o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@21a4891f{/solr,file:///tmp/src/solr/solr/solrj/build/tmp/tests-cwd/,AVAILABLE}
  2> 4919 INFO  (jetty-launcher-8-thread-1) [] o.e.j.s.AbstractConnector 
Started ServerConnector@7496c6b6{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:36981}
  2> 4923 INFO  (jetty-launcher-8-thread-1) [] o.e.j.s.Server Started 
Server@72070f39{STARTING}[10.0.16,sto=0] @5000ms
  2> 4947 ERROR (jetty-launcher-8-thread-1) [] o.a.s.u.StartupLoggingUtils 
Missing Java Option solr.log.dir. Logging may be missing or incomplete.
  2> 4949 INFO  (jetty-launcher-8-thread-1) [] o.a.s.s.CoreContainerProvider 
Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
  2> 4961 INFO  (jetty-launcher-8-thread-1) [] o.a.s.s.CoreContainerProvider  
___      _       Welcome to Apache Solr™ version 9.4.0-SNAPSHOT 
5cc252fb39f138f21e5894d08ba987479c90193a [snapshot build, details omitted]
  2> 4964 INFO  (jetty-launcher-8-thread-1) [] o.a.s.s.CoreContainerProvider / 
__| ___| |_ _   Starting in cloud mode on port null
  2> 4965 INFO  (jetty-launcher-8-thread-1) [] o.a.s.s.CoreContainerProvider 
\__ \/ _ \ | '_|  Install dir: /tmp/src/solr/solr
  2> 4966 INFO  (jetty-launcher-8-thread-1) [] o.a.s.s.CoreContainerProvider 
|___/\___/_|_|    Start time: 2023-09-10T17:40:42.094478Z
  2> 4982 INFO  (jetty-launcher-8-thread-1) [] o.a.s.s.CoreContainerProvider 
Solr Home: 
/tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.request.json.DirectJsonQueryRequestFacetingIntegrationTest_FBE234975B28C38F-001/tempDir-001/node1
 (source: servlet config: solr.solr.home)
  2> 5005 WARN  (jetty-launcher-8-thread-1) [] o.a.s.c.c.SolrZkClient Using 
default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates 
an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper 
nodes
  2> 5049 INFO  (jetty-launcher-8-thread-1) [] o.a.s.c.c.ConnectionManager 
Waiting up to 30000ms for client to connect to ZooKeeper
  2> 5068 INFO  (zkConnectionManagerCallback-10-thread-1) [] 
o.a.s.c.c.ConnectionManager zkClient has connected
  2> 5069 INFO  (jetty-launcher-8-thread-1) [] o.a.s.c.c.ConnectionManager 
Client is connected to ZooKeeper
  2> 5072 WARN  (jetty-launcher-8-thread-1) [] o.a.s.c.c.SolrZkClient Using 
default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 
'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 5088 INFO  (jetty-launcher-8-thread-1) [] o.a.s.c.NodeConfig solr.xml 
found in ZooKeeper. Loading...
  2> 5090 WARN  (jetty-launcher-8-thread-1) [] o.a.s.D.solrxml-zookeeper 
Loading solr.xml from zookeeper is deprecated. See reference guide for details.
  2> 5241 INFO  (jetty-launcher-8-thread-1) [] o.a.s.c.SolrXmlConfig Metrics 
collection is disabled.
  2> 6951 INFO  (jetty-launcher-8-thread-1) [] o.a.s.c.ZkContainer Zookeeper 
client=127.0.0.1:40721/solr
  2> 6989 INFO  (jetty-launcher-8-thread-1) [] 
o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater 
with useDistributedStateUpdate=true. Solr will be using distributed cluster 
state updates.
  2> 7017 INFO  (jetty-launcher-8-thread-1) [n:127.0.0.1:36981_solr] 
o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to 
ZooKeeper
  2> 7044 INFO  (zkConnectionManagerCallback-20-thread-1) [] 
o.a.s.c.c.ConnectionManager zkClient has connected
  2> 7046 INFO  (jetty-launcher-8-thread-1) [n:127.0.0.1:36981_solr] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 7311 WARN  (jetty-launcher-8-thread-1) [n:127.0.0.1:36981_solr] 
o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; 
consider setting up ACLs as described in 
https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
  2> 7415 INFO  (jetty-launcher-8-thread-1) [n:127.0.0.1:36981_solr] 
o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater 
with useDistributedStateUpdate=true. Solr will be using distributed cluster 
state updates.
  2> 7458 INFO  (jetty-launcher-8-thread-1) [n:127.0.0.1:36981_solr] 
o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:36981_solr
  2> 7472 INFO  (jetty-launcher-8-thread-1) [n:127.0.0.1:36981_solr] 
o.a.s.c.Overseer Overseer 
(id=72057614616494084-127.0.0.1:36981_solr-n_0000000000) starting
  2> 7688 INFO  (jetty-launcher-8-thread-1) [n:127.0.0.1:36981_solr] 
o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:36981_solr
  2> 7690 INFO  
(OverseerStateUpdate-72057614616494084-127.0.0.1:36981_solr-n_0000000000) 
[n:127.0.0.1:36981_solr] o.a.s.c.Overseer Starting to work on the main queue : 
127.0.0.1:36981_solr
  2> 7731 INFO  (zkCallback-19-thread-1) [] o.a.s.c.c.ZkStateReader Updated 
live nodes from ZooKeeper... (0) -> (1)
  2> 7808 WARN  (jetty-launcher-8-thread-1) [n:127.0.0.1:36981_solr] 
o.a.s.c.CoreContainer Not all security plugins configured!  
authentication=disabled authorization=disabled.  Solr is only as secure as you 
make it. Consider configuring authentication/authorization before exposing Solr 
to users internal or external.  See https://s.apache.org/solrsecurity for more 
info
  2> 8068 INFO  (jetty-launcher-8-thread-1) [n:127.0.0.1:36981_solr] 
o.a.s.c.a.c.DistributedCollectionConfigSetCommandRunner Creating 
DistributedCollectionConfigSetCommandRunner. Collection and ConfigSet APIs are 
running distributed (not Overseer based)
  2> 8069 INFO  (jetty-launcher-8-thread-1) [n:127.0.0.1:36981_solr] 
o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater 
with useDistributedStateUpdate=true. Solr will be using distributed cluster 
state updates.
  2> 8662 INFO  (jetty-launcher-8-thread-1) [n:127.0.0.1:36981_solr] 
o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath 
/tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.request.json.DirectJsonQueryRequestFacetingIntegrationTest_FBE234975B28C38F-001/tempDir-001/node1
  2> 9959 INFO  (jetty-launcher-8-thread-1) [n:127.0.0.1:36981_solr] 
o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
  2> 10270 INFO  (jetty-launcher-8-thread-1) [n:127.0.0.1:36981_solr] 
o.a.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=36981, 
zkHost=127.0.0.1:40721/solr}
  2> Sep 10, 2023 4:40:39 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
  2> WARNING: Suite execution timed out: 
org.apache.solr.client.solrj.request.json.DirectJsonQueryRequestFacetingIntegrationTest
  2> ==== jstack at approximately timeout time ====
  2> "Test worker" ID=1 WAITING on 
com.carrotsearch.randomizedtesting.RandomizedRunner$2@6cd85c48
  2>    at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2>    - waiting on 
com.carrotsearch.randomizedtesting.RandomizedRunner$2@6cd85c48
  2>    at java.base@11.0.16/java.lang.Thread.join(Thread.java:1300)
  2>    at java.base@11.0.16/java.lang.Thread.join(Thread.java:1375)
  2>    at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:647)
  2>    at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:504)
  2>    at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:108)
  2>    at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:57)
  2>    at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:39)
  2>    at 
org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
  2>    at 
org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:52)
  2>    at 
java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
  2>    at 
java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2>    at 
java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2>    at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2>    at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
  2>    at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
  2>    at 
org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
  2>    at 
org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
  2>    at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
  2>    at 
org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176)
  2>    at 
org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
  2>    at 
org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
  2>    at 
org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
  2>    at 
org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
  2>    at 
org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
  2>    at 
org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
  2>    at 
app//worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
  2>    at 
app//worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
  2> 
  2> "Reference Handler" ID=2 RUNNABLE
  2>    at 
java.base@11.0.16/java.lang.ref.Reference.waitForReferencePendingList(Native 
Method)
  2>    at 
java.base@11.0.16/java.lang.ref.Reference.processPendingReferences(Reference.java:241)
  2>    at 
java.base@11.0.16/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:213)
  2> 
  2> "Finalizer" ID=3 WAITING on java.lang.ref.ReferenceQueue$Lock@7f768605
  2>    at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2>    - waiting on java.lang.ref.ReferenceQueue$Lock@7f768605
  2>    at 
java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2>    at 
java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:176)
  2>    at 
java.base@11.0.16/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:170)
  2> 
  2> "Signal Dispatcher" ID=4 RUNNABLE
  2> 
  2> "Common-Cleaner" ID=9 TIMED_WAITING on 
java.lang.ref.ReferenceQueue$Lock@3e61f36b
  2>    at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2>    - timed waiting on java.lang.ref.ReferenceQueue$Lock@3e61f36b
  2>    at 
java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2>    at 
java.base@11.0.16/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:148)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    at 
java.base@11.0.16/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:161)
  2> 
  2> "/127.0.0.1:42320 to /127.0.0.1:34909 workers" ID=11 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3d402564
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3d402564
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
org.gradle.internal.remote.internal.hub.queue.EndPointQueue.take(EndPointQueue.java:49)
  2>    at 
org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:403)
  2>    at 
org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2>    at 
org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    Locked synchronizers:
  2>    - java.util.concurrent.ThreadPoolExecutor$Worker@71809907
  2> 
  2> "/127.0.0.1:42320 to /127.0.0.1:34909 workers Thread 2" ID=12 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1e7debfc
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1e7debfc
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
org.gradle.internal.remote.internal.hub.queue.EndPointQueue.take(EndPointQueue.java:49)
  2>    at 
org.gradle.internal.remote.internal.hub.MessageHub$ConnectionDispatch.run(MessageHub.java:322)
  2>    at 
org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2>    at 
org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    Locked synchronizers:
  2>    - java.util.concurrent.ThreadPoolExecutor$Worker@35d019a3
  2> 
  2> "/127.0.0.1:42320 to /127.0.0.1:34909 workers Thread 3" ID=13 RUNNABLE (in 
native code)
  2>    at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>    - locked sun.nio.ch.Util$2@55701098
  2>    - locked sun.nio.ch.EPollSelectorImpl@64bb5bf6
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>    at 
org.gradle.internal.remote.internal.inet.SocketConnection$SocketInputStream.read(SocketConnection.java:185)
  2>    at com.esotericsoftware.kryo.io.Input.fill(Input.java:146)
  2>    at com.esotericsoftware.kryo.io.Input.require(Input.java:178)
  2>    at com.esotericsoftware.kryo.io.Input.readByte(Input.java:295)
  2>    at 
org.gradle.internal.serialize.kryo.KryoBackedDecoder.readByte(KryoBackedDecoder.java:88)
  2>    at 
org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:64)
  2>    at 
org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:52)
  2>    at 
org.gradle.internal.remote.internal.inet.SocketConnection.receive(SocketConnection.java:81)
  2>    at 
org.gradle.internal.remote.internal.hub.MessageHub$ConnectionReceive.run(MessageHub.java:270)
  2>    at 
org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2>    at 
org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    Locked synchronizers:
  2>    - java.util.concurrent.ThreadPoolExecutor$Worker@4c371370
  2> 
  2> 
"SUITE-DirectJsonQueryRequestFacetingIntegrationTest-seed#[FBE234975B28C38F]" 
ID=14 RUNNABLE
  2>    at 
java.management@11.0.16/sun.management.ThreadImpl.dumpThreads0(Native Method)
  2>    at 
java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:521)
  2>    at 
java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:509)
  2>    at 
app//com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:708)
  2>    at 
app//com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:59)
  2>    at 
app//com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:437)
  2>    - locked java.lang.Object@58ec2900
  2>    at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:716)
  2>    at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:138)
  2>    at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:637)
  2> 
  2> 
"SUITE-DirectJsonQueryRequestFacetingIntegrationTest-seed#[FBE234975B28C38F]-worker"
 ID=15 WAITING on java.util.concurrent.FutureTask@29bf1bea
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on java.util.concurrent.FutureTask@29bf1bea
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.FutureTask.awaitDone(FutureTask.java:447)
  2>    at 
java.base@11.0.16/java.util.concurrent.FutureTask.get(FutureTask.java:190)
  2>    at 
java.base@11.0.16/java.util.concurrent.AbstractExecutorService.invokeAll(AbstractExecutorService.java:247)
  2>    at 
app//org.apache.solr.cloud.MiniSolrCloudCluster.<init>(MiniSolrCloudCluster.java:380)
  2>    at 
app//org.apache.solr.cloud.MiniSolrCloudCluster$Builder.build(MiniSolrCloudCluster.java:1311)
  2>    at 
app//org.apache.solr.cloud.MiniSolrCloudCluster$Builder.configure(MiniSolrCloudCluster.java:1278)
  2>    at 
app//org.apache.solr.client.solrj.request.json.DirectJsonQueryRequestFacetingIntegrationTest.setupCluster(DirectJsonQueryRequestFacetingIntegrationTest.java:52)
  2>    at 
java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
  2>    at 
java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2>    at 
java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2>    at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2>    at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  2>    at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:886)
  2>    at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  2>    at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>    at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>    at 
app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>    at 
app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>    at 
app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2>    at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>    at 
app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>    at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>    at 
app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  2>    at 
app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>    at 
app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>    at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>    at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>    at 
app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2>    at 
app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>    at 
app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2>    at 
app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2>    at 
app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  2>    at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>    at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>    at 
app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2>    at 
app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  2>    at 
app//com.carrotsearch.randomizedtesting.ThreadLeakControl$$Lambda$186/0x0000000840222040.run(Unknown
 Source)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "ZkTestServer Run Thread" ID=16 WAITING on 
org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread@7c1e2a95
  2>    at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2>    - waiting on 
org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread@7c1e2a95
  2>    at java.base@11.0.16/java.lang.Thread.join(Thread.java:1300)
  2>    at java.base@11.0.16/java.lang.Thread.join(Thread.java:1375)
  2>    at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:919)
  2>    at 
app//org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:337)
  2>    at app//org.apache.solr.cloud.ZkTestServer$1.run(ZkTestServer.java:581)
  2> 
  2> "NIOServerCxnFactory.SelectorThread-0" ID=18 RUNNABLE (in native code)
  2>    at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>    - locked sun.nio.ch.Util$2@4244ba0b
  2>    - locked sun.nio.ch.EPollSelectorImpl@7b592f6a
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>    at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.select(NIOServerCnxnFactory.java:403)
  2>    at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.run(NIOServerCnxnFactory.java:368)
  2> 
  2> "NIOServerCxnFactory.SelectorThread-1" ID=19 RUNNABLE (in native code)
  2>    at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>    - locked sun.nio.ch.Util$2@2a680ba0
  2>    - locked sun.nio.ch.EPollSelectorImpl@272fc6fa
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>    at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.select(NIOServerCnxnFactory.java:403)
  2>    at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.run(NIOServerCnxnFactory.java:368)
  2> 
  2> "NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:0" ID=20 RUNNABLE (in 
native code)
  2>    at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>    - locked sun.nio.ch.Util$2@405cb4f6
  2>    - locked sun.nio.ch.EPollSelectorImpl@72ccf1e8
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>    at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread.select(NIOServerCnxnFactory.java:205)
  2>    at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread.run(NIOServerCnxnFactory.java:181)
  2> 
  2> "ConnnectionExpirer" ID=17 TIMED_WAITING
  2>    at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2>    at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory$ConnectionExpirerThread.run(NIOServerCnxnFactory.java:554)
  2> 
  2> "NIOWorkerThread-1" ID=21 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1893c11a
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1893c11a
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "SessionTracker" ID=22 TIMED_WAITING
  2>    at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2>    at 
app//org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:163)
  2> 
  2> "SyncThread:0" ID=23 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@41d2cce4
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@41d2cce4
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>    at 
app//org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:170)
  2> 
  2> "ProcessThread(sid:0 cport:40721):" ID=24 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@640c3b2c
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@640c3b2c
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>    at 
app//org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:142)
  2> 
  2> "RequestThrottler" ID=25 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@715ac5f4
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@715ac5f4
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>    at 
app//org.apache.zookeeper.server.RequestThrottler.run(RequestThrottler.java:147)
  2> 
  2> "NIOWorkerThread-2" ID=26 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1893c11a
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1893c11a
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> 
"SUITE-DirectJsonQueryRequestFacetingIntegrationTest-seed#[FBE234975B28C38F]-worker-SendThread(127.0.0.1:40721)"
 ID=27 RUNNABLE (in native code)
  2>    at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>    - locked sun.nio.ch.Util$2@774741d4
  2>    - locked sun.nio.ch.EPollSelectorImpl@20250634
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
  2>    at 
app//org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:332)
  2>    at 
app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1274)
  2> 
  2> 
"SUITE-DirectJsonQueryRequestFacetingIntegrationTest-seed#[FBE234975B28C38F]-worker-EventThread"
 ID=28 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@25f58c47
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@25f58c47
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>    at 
app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:534)
  2> 
  2> "NIOWorkerThread-3" ID=29 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1893c11a
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1893c11a
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "zkConnectionManagerCallback-3-thread-1" ID=30 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2279868
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2279868
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> 
"SUITE-DirectJsonQueryRequestFacetingIntegrationTest-seed#[FBE234975B28C38F]-worker-SendThread(127.0.0.1:40721)"
 ID=31 RUNNABLE (in native code)
  2>    at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>    - locked sun.nio.ch.Util$2@4fa32019
  2>    - locked sun.nio.ch.EPollSelectorImpl@47bf9b7f
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
  2>    at 
app//org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:332)
  2>    at 
app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1274)
  2> 
  2> 
"SUITE-DirectJsonQueryRequestFacetingIntegrationTest-seed#[FBE234975B28C38F]-worker-EventThread"
 ID=32 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4ea1b07b
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4ea1b07b
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>    at 
app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:534)
  2> 
  2> "zkConnectionManagerCallback-5-thread-1" ID=33 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@70fb951f
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@70fb951f
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "jetty-launcher-8-thread-1" ID=37 WAITING on 
java.util.concurrent.CountDownLatch$Sync@314d9f1c
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on java.util.concurrent.CountDownLatch$Sync@314d9f1c
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
  2>    at 
java.base@11.0.16/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
  2>    at 
app//org.apache.solr.servlet.CoreContainerProvider$ContextInitializationKey.waitForReadyService(CoreContainerProvider.java:525)
  2>    at 
app//org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:564)
  2>    at 
app//org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
  2>    at 
app//org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133)
  2>    at 
app//org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774)
  2>    at 
app//org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472)
  2>    at 
app//org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992)
  2>    at 
app//org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480)
  2>    at 
app//org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427)
  2>    - locked org.apache.solr.embedded.JettySolrRunner$1@260ed0f1
  2>    at 
app//org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253)
  2>    at 
app//org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94)
  2>    at 
app//org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616)
  2>    at 
app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554)
  2>    - locked org.apache.solr.embedded.JettySolrRunner@1dd1d32c
  2>    at 
app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525)
  2>    at 
app//org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:569)
  2>    at 
app//org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:375)
  2>    at 
app//org.apache.solr.cloud.MiniSolrCloudCluster$$Lambda$273/0x0000000840335c40.call(Unknown
 Source)
  2>    at 
java.base@11.0.16/java.util.concurrent.FutureTask.run(FutureTask.java:264)
  2>    at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>    at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$264/0x0000000840330440.run(Unknown
 Source)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    Locked synchronizers:
  2>    - java.util.concurrent.locks.ReentrantLock$NonfairSync@5e308a85
  2>    - java.util.concurrent.locks.ReentrantLock$NonfairSync@72cfbf31
  2>    - java.util.concurrent.locks.ReentrantLock$NonfairSync@221670c1
  2>    - java.util.concurrent.ThreadPoolExecutor$Worker@7df85721
  2> 
  2> "qtp803231303-39" ID=39 RUNNABLE (in native code)
  2>    at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>    - locked sun.nio.ch.Util$2@6bf34f12
  2>    - locked sun.nio.ch.EPollSelectorImpl@644a436a
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:455)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:248)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:193)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$317/0x000000084038d840.run(Unknown
 Source)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp803231303-40-acceptor-0@1d203eb-ServerConnector@7496c6b6{HTTP/1.1, 
(http/1.1, h2c)}{127.0.0.1:36981}" ID=40 RUNNABLE (in native code)
  2>    at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native 
Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
  2>    at 
java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
  2>    at 
app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
  2>    at 
app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    Locked synchronizers:
  2>    - java.util.concurrent.locks.ReentrantLock$NonfairSync@75cbb75a
  2> 
  2> "qtp803231303-41" ID=41 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@28a36380
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - timed waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@28a36380
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>    at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp803231303-42" ID=42 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@28a36380
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - timed waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@28a36380
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>    at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp803231303-43" ID=43 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@28a36380
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - timed waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@28a36380
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>    at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp803231303-44" ID=44 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@28a36380
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - timed waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@28a36380
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>    at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp803231303-45" ID=45 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@28a36380
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - timed waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@28a36380
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>    at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp803231303-46" ID=46 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@28a36380
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - timed waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@28a36380
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>    at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Session-HouseKeeper-1a3252d2-1" ID=47 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@59d93259
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - timed waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@59d93259
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>    at 
java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2>    at 
java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "httpShardExecutor-15-thread-1" ID=51 RUNNABLE (in native code)
  2>    at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>    - locked sun.nio.ch.Util$2@6f3e6b1b
  2>    - locked sun.nio.ch.EPollSelectorImpl@47e770c1
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:455)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:248)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:193)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$317/0x000000084038d840.run(Unknown
 Source)
  2>    at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>    at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$264/0x0000000840330440.run(Unknown
 Source)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    Locked synchronizers:
  2>    - java.util.concurrent.ThreadPoolExecutor$Worker@44ec3fbb
  2> 
  2> "httpShardExecutor-15-thread-2" ID=52 RUNNABLE (in native code)
  2>    at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>    - locked sun.nio.ch.Util$2@746bb4de
  2>    - locked sun.nio.ch.EPollSelectorImpl@64301bf0
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:455)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:248)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:193)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$317/0x000000084038d840.run(Unknown
 Source)
  2>    at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>    at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$264/0x0000000840330440.run(Unknown
 Source)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    Locked synchronizers:
  2>    - java.util.concurrent.ThreadPoolExecutor$Worker@fe282c3
  2> 
  2> "Connection evictor" ID=53 TIMED_WAITING
  2>    at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2>    at 
app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Connection evictor" ID=54 TIMED_WAITING
  2>    at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2>    at 
app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "h2sc-17-thread-1" ID=55 RUNNABLE (in native code)
  2>    at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>    - locked sun.nio.ch.Util$2@78cad2de
  2>    - locked sun.nio.ch.EPollSelectorImpl@457c47e
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:455)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:248)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:193)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$317/0x000000084038d840.run(Unknown
 Source)
  2>    at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>    at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$264/0x0000000840330440.run(Unknown
 Source)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    Locked synchronizers:
  2>    - java.util.concurrent.ThreadPoolExecutor$Worker@6ec0df28
  2> 
  2> "h2sc-17-thread-2" ID=56 RUNNABLE (in native code)
  2>    at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>    - locked sun.nio.ch.Util$2@2d335d6e
  2>    - locked sun.nio.ch.EPollSelectorImpl@49b82ca3
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:455)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:248)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:193)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$317/0x000000084038d840.run(Unknown
 Source)
  2>    at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>    at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$264/0x0000000840330440.run(Unknown
 Source)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    Locked synchronizers:
  2>    - java.util.concurrent.ThreadPoolExecutor$Worker@458e9210
  2> 
  2> "jetty-launcher-8-thread-1-SendThread(127.0.0.1:40721)" ID=57 RUNNABLE (in 
native code)
  2>    at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>    - locked sun.nio.ch.Util$2@67ca5b68
  2>    - locked sun.nio.ch.EPollSelectorImpl@5d8417db
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
  2>    at 
app//org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:332)
  2>    at 
app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1274)
  2> 
  2> "jetty-launcher-8-thread-1-EventThread" ID=58 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6aebe928
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6aebe928
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>    at 
app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:534)
  2> 
  2> "zkConnectionManagerCallback-20-thread-1" ID=59 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@62be1de2
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@62be1de2
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "OverseerStateUpdate-72057614616494084-127.0.0.1:36981_solr-n_0000000000" 
ID=60 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@579a1bc6
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - timed waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@579a1bc6
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>    at 
app//org.apache.solr.cloud.ZkDistributedQueue.peekElements(ZkDistributedQueue.java:488)
  2>    at 
app//org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:338)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> 
"OverseerCollectionConfigSetProcessor-72057614616494084-127.0.0.1:36981_solr-n_0000000000"
 ID=61 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2adde421
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - timed waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2adde421
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>    at 
app//org.apache.solr.cloud.ZkDistributedQueue.peekElements(ZkDistributedQueue.java:488)
  2>    at 
app//org.apache.solr.cloud.OverseerTaskQueue.peekTopN(OverseerTaskQueue.java:267)
  2>    at 
app//org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:272)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "CloserThread" ID=63 WAITING on java.lang.Object@1fcaeafb
  2>    at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2>    - waiting on java.lang.Object@1fcaeafb
  2>    at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
  2>    at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2599)
  2> 
  2> ^^==============================================
  2> 
  2> Sep 10, 2023 4:40:39 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> WARNING: Will linger awaiting termination of 37 leaked thread(s).
  2> Sep 10, 2023 4:40:40 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> SEVERE: 37 threads leaked from SUITE scope at 
org.apache.solr.client.solrj.request.json.DirectJsonQueryRequestFacetingIntegrationTest:
 
  2>    1) Thread[id=32, 
name=SUITE-DirectJsonQueryRequestFacetingIntegrationTest-seed#[FBE234975B28C38F]-worker-EventThread,
 state=WAITING, group=TGRP-DirectJsonQueryRequestFacetingIntegrationTest]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>         at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>         at 
app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:534)
  2>    2) Thread[id=40, 
name=qtp803231303-40-acceptor-0@1d203eb-ServerConnector@7496c6b6{HTTP/1.1, 
(http/1.1, h2c)}{127.0.0.1:36981}, state=RUNNABLE, 
group=TGRP-DirectJsonQueryRequestFacetingIntegrationTest]
  2>         at 
java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
  2>         at 
java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
  2>         at 
java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
  2>         at 
app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
  2>         at 
app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
  2>         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2>         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2>         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    3) Thread[id=31, 
name=SUITE-DirectJsonQueryRequestFacetingIntegrationTest-seed#[FBE234975B28C38F]-worker-SendThread(127.0.0.1:40721),
 state=RUNNABLE, group=TGRP-DirectJsonQueryRequestFacetingIntegrationTest]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
  2>         at 
app//org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:332)
  2>         at 
app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1274)
  2>    4) Thread[id=29, name=NIOWorkerThread-3, state=WAITING, 
group=TGRP-DirectJsonQueryRequestFacetingIntegrationTest]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>         at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>         at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    5) Thread[id=20, 
name=NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:0, state=RUNNABLE, 
group=TGRP-DirectJsonQueryRequestFacetingIntegrationTest]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread.select(NIOServerCnxnFactory.java:205)
  2>         at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread.run(NIOServerCnxnFactory.java:181)
  2>    6) Thread[id=25, name=RequestThrottler, state=WAITING, 
group=TGRP-DirectJsonQueryRequestFacetingIntegrationTest]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>         at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>         at 
app//org.apache.zookeeper.server.RequestThrottler.run(RequestThrottler.java:147)
  2>    7) Thread[id=19, name=NIOServerCxnFactory.SelectorThread-1, 
state=RUNNABLE, group=TGRP-DirectJsonQueryRequestFacetingIntegrationTest]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.select(NIOServerCnxnFactory.java:403)
  2>         at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.run(NIOServerCnxnFactory.java:368)
  2>    8) Thread[id=59, name=zkConnectionManagerCallback-20-thread-1, 
state=WAITING, group=TGRP-DirectJsonQueryRequestFacetingIntegrationTest]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>         at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>         at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    9) Thread[id=58, name=jetty-launcher-8-thread-1-EventThread, 
state=WAITING, group=TGRP-DirectJsonQueryRequestFacetingIntegrationTest]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>         at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>         at 
app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:534)
  2>   10) Thread[id=55, name=h2sc-17-thread-1, state=RUNNABLE, 
group=TGRP-DirectJsonQueryRequestFacetingIntegrationTest]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:455)
  2>         at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:248)
  2>         at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:193)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$317/0x000000084038d840.run(Unknown
 Source)
  2>         at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$264/0x0000000840330440.run(Unknown
 Source)
  2>         at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   11) Thread[id=18, name=NIOServerCxnFactory.SelectorThread-0, 
state=RUNNABLE, group=TGRP-DirectJsonQueryRequestFacetingIntegrationTest]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.select(NIOServerCnxnFactory.java:403)
  2>         at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.run(NIOServerCnxnFactory.java:368)
  2>   12) Thread[id=44, name=qtp803231303-44, state=TIMED_WAITING, 
group=TGRP-DirectJsonQueryRequestFacetingIntegrationTest]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   13) Thread[id=60, 
name=OverseerStateUpdate-72057614616494084-127.0.0.1:36981_solr-n_0000000000, 
state=TIMED_WAITING, group=Overseer state updater.]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at 
app//org.apache.solr.cloud.ZkDistributedQueue.peekElements(ZkDistributedQueue.java:488)
  2>         at 
app//org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:338)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   14) Thread[id=24, name=ProcessThread(sid:0 cport:40721):, state=WAITING, 
group=TGRP-DirectJsonQueryRequestFacetingIntegrationTest]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>         at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>         at 
app//org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:142)
  2>   15) Thread[id=42, name=qtp803231303-42, state=TIMED_WAITING, 
group=TGRP-DirectJsonQueryRequestFacetingIntegrationTest]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   16) Thread[id=52, name=httpShardExecutor-15-thread-2, state=RUNNABLE, 
group=TGRP-DirectJsonQueryRequestFacetingIntegrationTest]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecuti

[...truncated too long message...]

ava.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   27) Thread[id=56, name=h2sc-17-thread-2, state=RUNNABLE, 
group=TGRP-DirectJsonQueryRequestFacetingIntegrationTest]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:455)
  2>         at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:248)
  2>         at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:193)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$317/0x000000084038d840.run(Unknown
 Source)
  2>         at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$264/0x0000000840330440.run(Unknown
 Source)
  2>         at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   >     java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   >         at __randomizedtesting.SeedInfo.seed([FBE234975B28C38F]:0)
  2> 7206125 WARN  (NIOWorkerThread-1) [] o.a.z.s.NIOServerCnxn Close of 
session 0x1000004ca940001
  2>           => java.io.IOException: ZooKeeperServer not running
  2>    at 
org.apache.zookeeper.server.NIOServerCnxn.readLength(NIOServerCnxn.java:569)
  2> java.io.IOException: ZooKeeperServer not running
  2>    at 
org.apache.zookeeper.server.NIOServerCnxn.readLength(NIOServerCnxn.java:569) 
~[zookeeper-3.9.0.jar:3.9.0]
  2>    at 
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:349) 
[zookeeper-3.9.0.jar:3.9.0]
  2>    at 
org.apache.zookeeper.server.NIOServerCnxnFactory$IOWorkRequest.doWork(NIOServerCnxnFactory.java:508)
 [zookeeper-3.9.0.jar:3.9.0]
  2>    at 
org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:153)
 [zookeeper-3.9.0.jar:3.9.0]
  2>    at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 
[?:?]
  2>    at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 
[?:?]
  2>    at java.lang.Thread.run(Thread.java:829) [?:?]
  2> 7206126 WARN  
(SUITE-DirectJsonQueryRequestFacetingIntegrationTest-seed#[FBE234975B28C38F]-worker-SendThread(127.0.0.1:40721))
 [] o.a.z.ClientCnxn Session 0x1000004ca940001 for server 
localhost/127.0.0.1:40721, Closing socket connection. Attempting reconnect 
except it is a SessionExpiredException.
  2>           => EndOfStreamException: Unable to read additional data from 
server sessionid 0x1000004ca940001, likely server has closed socket
  2>    at 
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77)
  2> org.apache.zookeeper.ClientCnxn$EndOfStreamException: Unable to read 
additional data from server sessionid 0x1000004ca940001, likely server has 
closed socket
  2>    at 
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77) 
~[zookeeper-3.9.0.jar:3.9.0]
  2>    at 
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
 ~[zookeeper-3.9.0.jar:3.9.0]
  2>    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1274) 
[zookeeper-3.9.0.jar:3.9.0]
  2> NOTE: test params are: codec=Asserting(Lucene95): {}, docValues:{}, 
maxPointsInLeafNode=295, maxMBSortInHeap=5.083993187047896, 
sim=Asserting(RandomSimilarity(queryNorm=true): {}), locale=en-KN, 
timezone=America/Argentina/Ushuaia
  2> NOTE: Linux 5.4.0-100-generic amd64/Oracle Corporation 11.0.16 
(64-bit)/cpus=1,threads=33,free=285454112,total=343932928
  2> NOTE: All tests run in this JVM: 
[DirectJsonQueryRequestFacetingIntegrationTest]
  2> NOTE: reproduce with: gradlew test --tests 
DirectJsonQueryRequestFacetingIntegrationTest -Dtests.seed=FBE234975B28C38F 
-Dtests.locale=en-KN -Dtests.timezone=America/Argentina/Ushuaia 
-Dtests.asserts=true -Dtests.file.encoding=UTF-8

:solr:solrj:test (FAILURE): 1120 test(s), 1 failure(s), 40 skipped

1120 tests completed, 1 failed, 40 skipped

> Task :solr:solrj:test FAILED
> Task :solr:solrj:wipeTaskTemp

ERROR: The following test(s) have failed:
  - 
org.apache.solr.client.solrj.request.json.DirectJsonQueryRequestFacetingIntegrationTest.classMethod
 (:solr:solrj)
    Test output: 
/tmp/src/solr/solr/solrj/build/test-results/test/outputs/OUTPUT-org.apache.solr.client.solrj.request.json.DirectJsonQueryRequestFacetingIntegrationTest.txt
    Reproduce with: gradlew :solr:solrj:test --tests 
"org.apache.solr.client.solrj.request.json.DirectJsonQueryRequestFacetingIntegrationTest.classMethod"
 -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC 
-XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" 
-Ptests.seed=FBE234975B28C38F -Ptests.file.encoding=UTF-8

  - org.apache.solr.cloud.MultiThreadedOCPTest.classMethod (:solr:core)
    Test output: 
/tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.cloud.MultiThreadedOCPTest.txt
    Reproduce with: gradlew :solr:core:test --tests 
"org.apache.solr.cloud.MultiThreadedOCPTest.classMethod" -Ptests.jvms=96 
"-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC 
-XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" 
-Ptests.seed=FBE234975B28C38F -Ptests.file.encoding=UTF-8

  - org.apache.solr.cloud.MultiThreadedOCPTest.classMethod (:solr:core)
    Test output: 
/tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.cloud.MultiThreadedOCPTest.txt
    Reproduce with: gradlew :solr:core:test --tests 
"org.apache.solr.cloud.MultiThreadedOCPTest.classMethod" -Ptests.jvms=96 
"-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC 
-XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" 
-Ptests.seed=FBE234975B28C38F -Ptests.file.encoding=UTF-8


FAILURE: Build completed with 3 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:core:test'.
> There were failing tests. See the results at: 
> file:///tmp/src/solr/solr/core/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command 
> '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished 
> with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

3: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:solrj:test'.
> There were failing tests. See the results at: 
> file:///tmp/src/solr/solr/solrj/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 2h 2m 26s
654 actionable tasks: 637 executed, 17 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 2h2m48.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' 
'**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such 
file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 
23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscr...@solr.apache.org
For additional commands, e-mail: builds-h...@solr.apache.org