[JENKINS] Solr » Solr-Check-9.x - Build # 4365 - Still Failing!

[Apache Jenkins Server]

Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4365/

3 tests failed.
FAILED:  org.apache.solr.cloud.OverseerTest.testShardLeaderChange

Error Message:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=215, 
name=OverseerCollectionConfigSetProcessor-72057614712242188-127.0.0.1:46843_solr-n_0000000000,
 state=RUNNABLE, group=Overseer collection creation process.]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=215, 
name=OverseerCollectionConfigSetProcessor-72057614712242188-127.0.0.1:46843_solr-n_0000000000,
 state=RUNNABLE, group=Overseer collection creation process.]
        at 
__randomizedtesting.SeedInfo.seed([877B3D3D7C8CC18E:5928BACA6614347F]:0)
Caused by: org.apache.solr.common.AlreadyClosedException
        at __randomizedtesting.SeedInfo.seed([877B3D3D7C8CC18E]:0)
        at 
app//org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70)
        at 
app//org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:419)
        at 
app//org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:486)
        at 
app//org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:171)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)


FAILED:  org.apache.solr.cloud.TestLRUStatsCacheCloud.testBasicStats

Error Message:
java.lang.Exception: Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([877B3D3D7C8CC18E]:0)


FAILED:  org.apache.solr.cloud.TestLRUStatsCacheCloud.classMethod

Error Message:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([877B3D3D7C8CC18E]:0)




Build Log:
[...truncated 14031 lines...]
# ERROR: Error from server at 
http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2:
 org.apache.solr.cluster.placement.PlacementException: Not enough eligible 
nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME
ok 40 Random placement plugin using ENV in 22527ms
ok 41 post solr log into solr in 8963ms
ok 42 start solr with ssl in 20248ms
ok 43 start solr with ssl and auth in 22358ms
ok 44 SOLR-11740 check 'solr stop' connection in 28768ms
ok 45 status detects locally running solr in 14378ms
ok 46 status does not expose cli parameters to end user in 211ms
ok 47 listing out files in 2165ms
ok 48 copying files around in 6605ms
ok 49 upconfig in 2674ms

> Task :solr:packaging:integrationTests FAILED

> Task :solr:core:test

org.apache.solr.cloud.TestLRUStatsCacheCloud > testBasicStats FAILED
    java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([877B3D3D7C8CC18E]:0)

org.apache.solr.cloud.TestLRUStatsCacheCloud > classMethod FAILED
    java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([877B3D3D7C8CC18E]:0)

org.apache.solr.cloud.TestLRUStatsCacheCloud > test suite's output saved to 
/tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.cloud.TestLRUStatsCacheCloud.txt,
 copied below:
  2> 4989 INFO  (SUITE-TestLRUStatsCacheCloud-seed#[877B3D3D7C8CC18E]-worker) 
[] o.a.s.SolrTestCase Setting 'solr.default.confdir' system property to 
test-framework derived value of 
'/tmp/src/solr/solr/server/solr/configsets/_default/conf'
  2> 4991 INFO  (SUITE-TestLRUStatsCacheCloud-seed#[877B3D3D7C8CC18E]-worker) 
[] o.a.s.u.ErrorLogMuter Closing ErrorLogMuter-regex-2 after mutting 0 log 
messages
  2> 4991 INFO  (SUITE-TestLRUStatsCacheCloud-seed#[877B3D3D7C8CC18E]-worker) 
[] o.a.s.u.ErrorLogMuter Creating ErrorLogMuter-regex-3 for ERROR logs matching 
regex: ignore_exception
  2> 4993 INFO  (SUITE-TestLRUStatsCacheCloud-seed#[877B3D3D7C8CC18E]-worker) 
[] o.a.s.SolrTestCaseJ4 Created dataDir: 
/tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.cloud.TestLRUStatsCacheCloud_877B3D3D7C8CC18E-001/data-dir-2-001
  2> 4993 INFO  (SUITE-TestLRUStatsCacheCloud-seed#[877B3D3D7C8CC18E]-worker) 
[] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) 
w/NUMERIC_DOCVALUES_SYSPROP=false
  2> 4995 INFO  (SUITE-TestLRUStatsCacheCloud-seed#[877B3D3D7C8CC18E]-worker) 
[] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: 
@org.apache.solr.util.RandomizeSSL(reason="", value=0.0/0.0, ssl=0.0/0.0, 
clientAuth=0.0/0.0)
  2> 4995 INFO  (SUITE-TestLRUStatsCacheCloud-seed#[877B3D3D7C8CC18E]-worker) 
[] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: 
test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
  2> 5006 INFO  
(TEST-TestLRUStatsCacheCloud.testBasicStats-seed#[877B3D3D7C8CC18E]) [] 
o.a.s.SolrTestCaseJ4 ###Starting testBasicStats
  2> 5012 INFO  
(TEST-TestLRUStatsCacheCloud.testBasicStats-seed#[877B3D3D7C8CC18E]) [] 
o.a.s.SolrTestCaseJ4 ####initCore
  2> 5108 INFO  
(TEST-TestLRUStatsCacheCloud.testBasicStats-seed#[877B3D3D7C8CC18E]) [] 
o.a.s.c.SolrResourceLoader Added 2 libs to classloader, from paths: 
[/tmp/src/solr/solr/core/build/resources/test/solr/collection1/lib, 
/tmp/src/solr/solr/core/build/resources/test/solr/collection1/lib/classes]
  2> 5390 WARN  
(TEST-TestLRUStatsCacheCloud.testBasicStats-seed#[877B3D3D7C8CC18E]) [] 
o.a.s.c.SolrConfig You should not use LATEST as luceneMatchVersion property: if 
you use this setting, and then Solr upgrades to a newer release of Lucene, 
sizable changes may happen. If precise back compatibility is important then you 
should instead explicitly specify an actual Lucene version.
  2> 5391 INFO  
(TEST-TestLRUStatsCacheCloud.testBasicStats-seed#[877B3D3D7C8CC18E]) [] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.5.0
  2> 5462 INFO  
(TEST-TestLRUStatsCacheCloud.testBasicStats-seed#[877B3D3D7C8CC18E]) [] 
o.a.s.s.IndexSchema Schema name=tiny
  2> 5484 INFO  
(TEST-TestLRUStatsCacheCloud.testBasicStats-seed#[877B3D3D7C8CC18E]) [] 
o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
  2> 7606 WARN  
(TEST-TestLRUStatsCacheCloud.testBasicStats-seed#[877B3D3D7C8CC18E]) [] 
o.a.s.c.CoreContainer Not all security plugins configured!  
authentication=disabled authorization=disabled.  Solr is only as secure as you 
make it. Consider configuring authentication/authorization before exposing Solr 
to users internal or external.  See https://s.apache.org/solrsecurity for more 
info
  2> 7956 INFO  (coreLoadExecutor-18-thread-1) [ x:collection1] 
o.a.s.c.SolrResourceLoader Added 2 libs to classloader, from paths: 
[/tmp/src/solr/solr/core/build/resources/test/solr/collection1/lib, 
/tmp/src/solr/solr/core/build/resources/test/solr/collection1/lib/classes]
  2> 8011 INFO  (coreLoadExecutor-18-thread-1) [ x:collection1] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.5.0
  2> 8026 INFO  (coreLoadExecutor-18-thread-1) [ x:collection1] 
o.a.s.s.IndexSchema Schema name=tiny
  2> 8028 INFO  (coreLoadExecutor-18-thread-1) [ x:collection1] 
o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
  2> 8029 INFO  (coreLoadExecutor-18-thread-1) [ x:collection1] 
o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from 
instancedir /tmp/src/solr/solr/core/build/resources/test/solr/collection1, 
trusted=true
  2> 8093 INFO  (coreLoadExecutor-18-thread-1) [ x:collection1] 
o.a.s.c.SolrCore Opening new SolrCore at 
[/tmp/src/solr/solr/core/build/resources/test/solr/collection1], 
dataDir=[/tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.cloud.TestLRUStatsCacheCloud_877B3D3D7C8CC18E-001/data-dir-2-001/]
  2> 8139 INFO  (coreLoadExecutor-18-thread-1) [ x:collection1] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: 
minMergeSize=1000, mergeFactor=10, maxMergeSize=9223372036854775807, 
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, 
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022208E12, 
noCFSRatio=0.8036543503713172]
  2> 9954 INFO  (coreLoadExecutor-18-thread-1) [ x:collection1] 
o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
  2> 10097 INFO  (coreLoadExecutor-18-thread-1) [ x:collection1] 
o.a.s.u.CommitTracker Hard AutoCommit: disabled
  2> 10098 INFO  (coreLoadExecutor-18-thread-1) [ x:collection1] 
o.a.s.u.CommitTracker Soft AutoCommit: disabled
  2> 10103 INFO  (coreLoadExecutor-18-thread-1) [ x:collection1] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: 
maxMergeAtOnce=25, maxMergedSegmentMB=55.19551181793213, 
floorSegmentMB=1.1870613098144531, 
forceMergeDeletesPctAllowed=15.273915046866405, segmentsPerTier=32.0, 
maxCFSSegmentSizeMB=8.796093022208E12, noCFSRatio=1.0, 
deletesPctAllowed=47.02006987376987
  2> 10175 WARN  (coreLoadExecutor-18-thread-1) [ x:collection1] 
o.a.s.r.ManagedResourceStorage Cannot write to config directory 
/tmp/src/solr/solr/core/build/resources/test/solr/collection1/conf ; switching 
to use InMemory storage instead.
  2> 10205 INFO  (coreLoadExecutor-18-thread-1) [ x:collection1] 
o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
  2> 10228 INFO  (searcherExecutor-20-thread-1-processing-collection1) [ 
x:collection1] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
  2> 10482 INFO  
(TEST-TestLRUStatsCacheCloud.testBasicStats-seed#[877B3D3D7C8CC18E]) [] 
o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
  2> 10562 INFO  
(TEST-TestLRUStatsCacheCloud.testBasicStats-seed#[877B3D3D7C8CC18E]) [] 
o.a.s.SolrTestCaseJ4 ####initCore end
  2> 10580 INFO  
(TEST-TestLRUStatsCacheCloud.testBasicStats-seed#[877B3D3D7C8CC18E]) [] 
o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in 
/tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.cloud.TestLRUStatsCacheCloud_877B3D3D7C8CC18E-001/tempDir-001
  2> 10585 INFO  
(TEST-TestLRUStatsCacheCloud.testBasicStats-seed#[877B3D3D7C8CC18E]) [] 
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
  2> 10598 INFO  (ZkTestServer Run Thread) [] o.a.s.c.ZkTestServer client port: 
0.0.0.0/0.0.0.0:0
  2> 10599 INFO  (ZkTestServer Run Thread) [] o.a.s.c.ZkTestServer Starting 
server
  2> 10603 WARN  (ZkTestServer Run Thread) [] o.a.z.s.ServerCnxnFactory 
maxCnxns is not configured, using default value 0.
  2> 10698 INFO  
(TEST-TestLRUStatsCacheCloud.testBasicStats-seed#[877B3D3D7C8CC18E]) [] 
o.a.s.c.ZkTestServer start zk server on port: 41155
  2> 10704 WARN  
(TEST-TestLRUStatsCacheCloud.testBasicStats-seed#[877B3D3D7C8CC18E]) [] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. 
ZkCredentialsInjector is not secure, it creates an empty list of credentials 
which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 10721 INFO  
(TEST-TestLRUStatsCacheCloud.testBasicStats-seed#[877B3D3D7C8CC18E]) [] 
o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to 
ZooKeeper
  2> 10754 INFO  (zkConnectionManagerCallback-25-thread-1) [] 
o.a.s.c.c.ConnectionManager zkClient has connected
  2> 10754 INFO  
(TEST-TestLRUStatsCacheCloud.testBasicStats-seed#[877B3D3D7C8CC18E]) [] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 10754 WARN  
(TEST-TestLRUStatsCacheCloud.testBasicStats-seed#[877B3D3D7C8CC18E]) [] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not 
secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 10769 WARN  
(TEST-TestLRUStatsCacheCloud.testBasicStats-seed#[877B3D3D7C8CC18E]) [] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. 
ZkCredentialsInjector is not secure, it creates an empty list of credentials 
which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 10792 INFO  
(TEST-TestLRUStatsCacheCloud.testBasicStats-seed#[877B3D3D7C8CC18E]) [] 
o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to 
ZooKeeper
  2> 10803 INFO  (zkConnectionManagerCallback-27-thread-1) [] 
o.a.s.c.c.ConnectionManager zkClient has connected
  2> 10804 INFO  
(TEST-TestLRUStatsCacheCloud.testBasicStats-seed#[877B3D3D7C8CC18E]) [] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 10804 WARN  
(TEST-TestLRUStatsCacheCloud.testBasicStats-seed#[877B3D3D7C8CC18E]) [] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not 
secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 10804 WARN  
(TEST-TestLRUStatsCacheCloud.testBasicStats-seed#[877B3D3D7C8CC18E]) [] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. 
ZkCredentialsInjector is not secure, it creates an empty list of credentials 
which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 10828 INFO  
(TEST-TestLRUStatsCacheCloud.testBasicStats-seed#[877B3D3D7C8CC18E]) [] 
o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to 
ZooKeeper
  2> 10858 INFO  (zkConnectionManagerCallback-29-thread-1) [] 
o.a.s.c.c.ConnectionManager zkClient has connected
  2> 10858 INFO  
(TEST-TestLRUStatsCacheCloud.testBasicStats-seed#[877B3D3D7C8CC18E]) [] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 10859 WARN  
(TEST-TestLRUStatsCacheCloud.testBasicStats-seed#[877B3D3D7C8CC18E]) [] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not 
secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 11191 INFO  (jetty-launcher-30-thread-2) [] o.a.s.e.JettySolrRunner Start 
Jetty (configured port=0, binding port=0)
  2> 11192 INFO  (jetty-launcher-30-thread-2) [] o.a.s.e.JettySolrRunner Trying 
to start Jetty on port 0 try number 2 ...
  2> 11208 INFO  (jetty-launcher-30-thread-1) [] o.a.s.e.JettySolrRunner Start 
Jetty (configured port=0, binding port=0)
  2> 11209 INFO  (jetty-launcher-30-thread-1) [] o.a.s.e.JettySolrRunner Trying 
to start Jetty on port 0 try number 2 ...
  2> 11210 INFO  (jetty-launcher-30-thread-2) [] o.e.j.s.Server jetty-10.0.14; 
built: 2023-02-22T23:12:32.272Z; git: 976721d0f3e903a243584d47870ad2f2c1bf9e55; 
jvm 11.0.16+8
  2> 11212 INFO  (jetty-launcher-30-thread-1) [] o.e.j.s.Server jetty-10.0.14; 
built: 2023-02-22T23:12:32.272Z; git: 976721d0f3e903a243584d47870ad2f2c1bf9e55; 
jvm 11.0.16+8
  2> 11276 INFO  (jetty-launcher-30-thread-1) [] 
o.e.j.s.s.DefaultSessionIdManager Session workerName=node0
  2> 11293 INFO  (jetty-launcher-30-thread-2) [] 
o.e.j.s.s.DefaultSessionIdManager Session workerName=node0
  2> 11336 INFO  (jetty-launcher-30-thread-2) [] o.e.j.s.h.ContextHandler 
Started 
o.e.j.s.ServletContextHandler@1564c459{/solr,file:///tmp/src/solr/solr/core/build/tmp/tests-cwd/,AVAILABLE}
  2> 11343 INFO  (jetty-launcher-30-thread-1) [] o.e.j.s.h.ContextHandler 
Started 
o.e.j.s.ServletContextHandler@14669e14{/solr,file:///tmp/src/solr/solr/core/build/tmp/tests-cwd/,AVAILABLE}
  2> 11345 INFO  (jetty-launcher-30-thread-1) [] o.e.j.s.AbstractConnector 
Started ServerConnector@3ab31b79{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:38599}
  2> 11345 INFO  (jetty-launcher-30-thread-2) [] o.e.j.s.AbstractConnector 
Started ServerConnector@2cf369e5{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:34605}
  2> 11348 INFO  (jetty-launcher-30-thread-2) [] o.e.j.s.Server Started 
Server@73fd6d3a{STARTING}[10.0.14,sto=0] @11372ms
  2> 11351 INFO  (jetty-launcher-30-thread-1) [] o.e.j.s.Server Started 
Server@5ad510cd{STARTING}[10.0.14,sto=0] @11373ms
  2> 11359 ERROR (jetty-launcher-30-thread-1) [] o.a.s.u.StartupLoggingUtils 
Missing Java Option solr.log.dir. Logging may be missing or incomplete.
  2> 11359 ERROR (jetty-launcher-30-thread-2) [] o.a.s.u.StartupLoggingUtils 
Missing Java Option solr.log.dir. Logging may be missing or incomplete.
  2> 11360 INFO  (jetty-launcher-30-thread-1) [] o.a.s.s.CoreContainerProvider 
Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
  2> 11360 INFO  (jetty-launcher-30-thread-2) [] o.a.s.s.CoreContainerProvider 
Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
  2> 11372 INFO  (jetty-launcher-30-thread-1) [] o.a.s.s.CoreContainerProvider  
___      _       Welcome to Apache Solr™ version 9.3.0
  2> 11371 INFO  (jetty-launcher-30-thread-2) [] o.a.s.s.CoreContainerProvider  
___      _       Welcome to Apache Solr™ version 9.3.0
  2> 11376 INFO  (jetty-launcher-30-thread-2) [] o.a.s.s.CoreContainerProvider 
/ __| ___| |_ _   Starting in cloud mode on port null
  2> 11376 INFO  (jetty-launcher-30-thread-2) [] o.a.s.s.CoreContainerProvider 
\__ \/ _ \ | '_|  Install dir: /tmp/src/solr/solr
  2> 11375 INFO  (jetty-launcher-30-thread-1) [] o.a.s.s.CoreContainerProvider 
/ __| ___| |_ _   Starting in cloud mode on port null
  2> 11378 INFO  (jetty-launcher-30-thread-1) [] o.a.s.s.CoreContainerProvider 
\__ \/ _ \ | '_|  Install dir: /tmp/src/solr/solr
  2> 11379 INFO  (jetty-launcher-30-thread-1) [] o.a.s.s.CoreContainerProvider 
|___/\___/_|_|    Start time: 2023-05-19T09:20:13.519011Z
  2> 11382 INFO  (jetty-launcher-30-thread-1) [] o.a.s.s.CoreContainerProvider 
Solr Home: 
/tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.cloud.TestLRUStatsCacheCloud_877B3D3D7C8CC18E-001/tempDir-001/node1
 (source: servlet config: solr.solr.home)
  2> 11383 WARN  (jetty-launcher-30-thread-1) [] o.a.s.c.c.SolrZkClient Using 
default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates 
an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper 
nodes
  2> 11377 INFO  (jetty-launcher-30-thread-2) [] o.a.s.s.CoreContainerProvider 
|___/\___/_|_|    Start time: 2023-05-19T09:20:13.517909Z
  2> 11384 INFO  (jetty-launcher-30-thread-2) [] o.a.s.s.CoreContainerProvider 
Solr Home: 
/tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.cloud.TestLRUStatsCacheCloud_877B3D3D7C8CC18E-001/tempDir-001/node2
 (source: servlet config: solr.solr.home)
  2> 11386 INFO  (jetty-launcher-30-thread-1) [] o.a.s.c.c.ConnectionManager 
Waiting up to 30000ms for client to connect to ZooKeeper
  2> 11385 WARN  (jetty-launcher-30-thread-2) [] o.a.s.c.c.SolrZkClient Using 
default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates 
an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper 
nodes
  2> 11434 INFO  (zkConnectionManagerCallback-32-thread-1) [] 
o.a.s.c.c.ConnectionManager zkClient has connected
  2> 11435 INFO  (jetty-launcher-30-thread-2) [] o.a.s.c.c.ConnectionManager 
Waiting up to 30000ms for client to connect to ZooKeeper
  2> 11446 INFO  (jetty-launcher-30-thread-1) [] o.a.s.c.c.ConnectionManager 
Client is connected to ZooKeeper
  2> 11446 INFO  (zkConnectionManagerCallback-34-thread-1) [] 
o.a.s.c.c.ConnectionManager zkClient has connected
  2> 11446 WARN  (jetty-launcher-30-thread-1) [] o.a.s.c.c.SolrZkClient Using 
default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 
'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 11447 INFO  (jetty-launcher-30-thread-2) [] o.a.s.c.c.ConnectionManager 
Client is connected to ZooKeeper
  2> 11448 WARN  (jetty-launcher-30-thread-2) [] o.a.s.c.c.SolrZkClient Using 
default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 
'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 11450 INFO  (jetty-launcher-30-thread-1) [] o.a.s.c.NodeConfig solr.xml 
found in ZooKeeper. Loading...
  2> 11452 WARN  (jetty-launcher-30-thread-1) [] o.a.s.D.solrxml-zookeeper 
Loading solr.xml from zookeeper is deprecated. See reference guide for details.
  2> 11458 INFO  (jetty-launcher-30-thread-2) [] o.a.s.c.NodeConfig solr.xml 
found in ZooKeeper. Loading...
  2> 11690 INFO  (jetty-launcher-30-thread-2) [] o.a.s.c.ZkContainer Zookeeper 
client=127.0.0.1:41155/solr
  2> 11690 INFO  (jetty-launcher-30-thread-1) [] o.a.s.c.ZkContainer Zookeeper 
client=127.0.0.1:41155/solr
  2> 11721 INFO  (jetty-launcher-30-thread-1) [] 
o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater 
with useDistributedStateUpdate=true. Solr will be using distributed cluster 
state updates.
  2> 11738 INFO  (jetty-launcher-30-thread-2) [] 
o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater 
with useDistributedStateUpdate=true. Solr will be using distributed cluster 
state updates.
  2> 11739 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:38599_solr] 
o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to 
ZooKeeper
  2> 11758 INFO  (jetty-launcher-30-thread-2) [n:127.0.0.1:34605_solr] 
o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to 
ZooKeeper
  2> 11776 INFO  (zkConnectionManagerCallback-52-thread-1) [] 
o.a.s.c.c.ConnectionManager zkClient has connected
  2> 11777 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:38599_solr] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 11781 INFO  (zkConnectionManagerCallback-54-thread-1) [] 
o.a.s.c.c.ConnectionManager zkClient has connected
  2> 11781 INFO  (jetty-launcher-30-thread-2) [n:127.0.0.1:34605_solr] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 11937 WARN  (jetty-launcher-30-thread-1) [n:127.0.0.1:38599_solr] 
o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; 
consider setting up ACLs as described in 
https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
  2> 11937 WARN  (jetty-launcher-30-thread-2) [n:127.0.0.1:34605_solr] 
o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; 
consider setting up ACLs as described in 
https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
  2> 11981 INFO  (jetty-launcher-30-thread-2) [n:127.0.0.1:34605_solr] 
o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater 
with useDistributedStateUpdate=true. Solr will be using distributed cluster 
state updates.
  2> 11981 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:38599_solr] 
o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater 
with useDistributedStateUpdate=true. Solr will be using distributed cluster 
state updates.
  2> 12016 INFO  (jetty-launcher-30-thread-2) [n:127.0.0.1:34605_solr] 
o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:34605_solr
  2> 12018 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:38599_solr] 
o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:38599_solr
  2> 12040 INFO  (jetty-launcher-30-thread-2) [n:127.0.0.1:34605_solr] 
o.a.s.c.Overseer Overseer 
(id=72057614050394118-127.0.0.1:34605_solr-n_0000000000) starting
  2> 12045 INFO  (zkCallback-53-thread-1) [] o.a.s.c.c.ZkStateReader Updated 
live nodes from ZooKeeper... (0) -> (1)
  2> 12073 INFO  (zkCallback-51-thread-1) [] o.a.s.c.c.ZkStateReader Updated 
live nodes from ZooKeeper... (0) -> (1)
  2> 12111 WARN  (jetty-launcher-30-thread-1) [n:127.0.0.1:38599_solr] 
o.a.s.c.CoreContainer Not all security plugins configured!  
authentication=disabled authorization=disabled.  Solr is only as secure as you 
make it. Consider configuring authentication/authorization before exposing Solr 
to users internal or external.  See https://s.apache.org/solrsecurity for more 
info
  2> 12222 INFO  
(OverseerStateUpdate-72057614050394118-127.0.0.1:34605_solr-n_0000000000) 
[n:127.0.0.1:34605_solr] o.a.s.c.Overseer Starting to work on the main queue : 
127.0.0.1:34605_solr
  2> 12232 INFO  (jetty-launcher-30-thread-2) [n:127.0.0.1:34605_solr] 
o.a.s.c.ZkController Publish node=127.0.0.1:34605_solr as DOWN
  2> 12239 INFO  (jetty-launcher-30-thread-2) [n:127.0.0.1:34605_solr] 
o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:34605_solr
  2> 12254 INFO  (zkCallback-53-thread-1) [] o.a.s.c.c.ZkStateReader Updated 
live nodes from ZooKeeper... (1) -> (2)
  2> 12254 INFO  (zkCallback-51-thread-1) [] o.a.s.c.c.ZkStateReader Updated 
live nodes from ZooKeeper... (1) -> (2)
  2> 12271 WARN  (jetty-launcher-30-thread-2) [n:127.0.0.1:34605_solr] 
o.a.s.c.CoreContainer Not all security plugins configured!  
authentication=disabled authorization=disabled.  Solr is only as secure as you 
make it. Consider configuring authentication/authorization before exposing Solr 
to users internal or external.  See https://s.apache.org/solrsecurity for more 
info
  2> 12379 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:38599_solr] 
o.a.s.c.a.c.DistributedCollectionConfigSetCommandRunner Creating 
DistributedCollectionConfigSetCommandRunner. Collection and ConfigSet APIs are 
running distributed (not Overseer based)
  2> 12382 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:38599_solr] 
o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater 
with useDistributedStateUpdate=true. Solr will be using distributed cluster 
state updates.
  2> 12695 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:38599_solr] 
o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath 
/tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.cloud.TestLRUStatsCacheCloud_877B3D3D7C8CC18E-001/tempDir-001/node1
  2> 12700 INFO  (jetty-launcher-30-thread-2) [n:127.0.0.1:34605_solr] 
o.a.s.c.a.c.DistributedCollectionConfigSetCommandRunner Creating 
DistributedCollectionConfigSetCommandRunner. Collection and ConfigSet APIs are 
running distributed (not Overseer based)
  2> 12701 INFO  (jetty-launcher-30-thread-2) [n:127.0.0.1:34605_solr] 
o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater 
with useDistributedStateUpdate=true. Solr will be using distributed cluster 
state updates.
  2> 12844 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:38599_solr] 
o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
  2> 12867 INFO  (jetty-launcher-30-thread-2) [n:127.0.0.1:34605_solr] 
o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath 
/tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.cloud.TestLRUStatsCacheCloud_877B3D3D7C8CC18E-001/tempDir-001/node2
  2> 12951 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:38599_solr] 
o.a.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=38599, 
zkHost=127.0.0.1:41155/solr}
  2> 13104 INFO  (jetty-launcher-30-thread-2) [n:127.0.0.1:34605_solr] 
o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
  2> 13155 INFO  (jetty-launcher-30-thread-2) [n:127.0.0.1:34605_solr] 
o.a.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=34605, 
zkHost=127.0.0.1:41155/solr}
  2> ਮਈ 19, 2023 8:20:07 ਪੂ.ਦੁ. 
com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
  2> WARNING: Suite execution timed out: 
org.apache.solr.cloud.TestLRUStatsCacheCloud
  2> ==== jstack at approximately timeout time ====
  2> "Test worker" ID=1 WAITING on 
com.carrotsearch.randomizedtesting.RandomizedRunner$2@7bc96710
  2>    at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2>    - waiting on 
com.carrotsearch.randomizedtesting.RandomizedRunner$2@7bc96710
  2>    at java.base@11.0.16/java.lang.Thread.join(Thread.java:1300)
  2>    at java.base@11.0.16/java.lang.Thread.join(Thread.java:1375)
  2>    at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:647)
  2>    at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:504)
  2>    at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:108)
  2>    at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:57)
  2>    at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:39)
  2>    at 
org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
  2>    at 
org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:52)
  2>    at 
java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
  2>    at 
java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2>    at 
java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2>    at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2>    at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
  2>    at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
  2>    at 
org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
  2>    at 
org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
  2>    at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
  2>    at 
org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176)
  2>    at 
org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
  2>    at 
org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
  2>    at 
org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
  2>    at 
org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
  2>    at 
org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
  2>    at 
org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
  2>    at 
app//worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
  2>    at 
app//worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
  2> 
  2> "Reference Handler" ID=2 RUNNABLE
  2>    at 
java.base@11.0.16/java.lang.ref.Reference.waitForReferencePendingList(Native 
Method)
  2>    at 
java.base@11.0.16/java.lang.ref.Reference.processPendingReferences(Reference.java:241)
  2>    at 
java.base@11.0.16/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:213)
  2> 
  2> "Finalizer" ID=3 WAITING on java.lang.ref.ReferenceQueue$Lock@3af46d2b
  2>    at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2>    - waiting on java.lang.ref.ReferenceQueue$Lock@3af46d2b
  2>    at 
java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2>    at 
java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:176)
  2>    at 
java.base@11.0.16/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:170)
  2> 
  2> "Signal Dispatcher" ID=4 RUNNABLE
  2> 
  2> "Common-Cleaner" ID=9 TIMED_WAITING on 
java.lang.ref.ReferenceQueue$Lock@3b45552b
  2>    at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2>    - timed waiting on java.lang.ref.ReferenceQueue$Lock@3b45552b
  2>    at 
java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2>    at 
java.base@11.0.16/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:148)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    at 
java.base@11.0.16/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:161)
  2> 
  2> "/127.0.0.1:57604 to /127.0.0.1:36981 workers" ID=11 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@435f3211
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@435f3211
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
java.base@11.0.16/java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:367)
  2>    at 
org.gradle.api.internal.tasks.testing.worker.TestWorker.submitToRun(TestWorker.java:207)
  2>    - locked org.gradle.api.internal.tasks.testing.worker.TestWorker@b0454b4
  2>    at 
org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:169)
  2>    at 
java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
  2>    at 
java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2>    at 
java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2>    at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2>    at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
  2>    at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
  2>    at 
org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
  2>    at 
org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
  2>    at 
org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
  2>    at 
org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2>    at 
org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    Locked synchronizers:
  2>    - java.util.concurrent.ThreadPoolExecutor$Worker@294425a7
  2> 
  2> "/127.0.0.1:57604 to /127.0.0.1:36981 workers Thread 2" ID=12 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5adb9264
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5adb9264
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
org.gradle.internal.remote.internal.hub.queue.EndPointQueue.take(EndPointQueue.java:49)
  2>    at 
org.gradle.internal.remote.internal.hub.MessageHub$ConnectionDispatch.run(MessageHub.java:322)
  2>    at 
org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2>    at 
org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    Locked synchronizers:
  2>    - java.util.concurrent.ThreadPoolExecutor$Worker@3270d194
  2> 
  2> "/127.0.0.1:57604 to /127.0.0.1:36981 workers Thread 3" ID=13 RUNNABLE (in 
native code)
  2>    at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>    - locked sun.nio.ch.Util$2@4ba67819
  2>    - locked sun.nio.ch.EPollSelectorImpl@622e342d
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>    at 
org.gradle.internal.remote.internal.inet.SocketConnection$SocketInputStream.read(SocketConnection.java:185)
  2>    at com.esotericsoftware.kryo.io.Input.fill(Input.java:146)
  2>    at com.esotericsoftware.kryo.io.Input.require(Input.java:178)
  2>    at com.esotericsoftware.kryo.io.Input.readByte(Input.java:295)
  2>    at 
org.gradle.internal.serialize.kryo.KryoBackedDecoder.readByte(KryoBackedDecoder.java:88)
  2>    at 
org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:64)
  2>    at 
org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:52)
  2>    at 
org.gradle.internal.remote.internal.inet.SocketConnection.receive(SocketConnection.java:81)
  2>    at 
org.gradle.internal.remote.internal.hub.MessageHub$ConnectionReceive.run(MessageHub.java:270)
  2>    at 
org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2>    at 
org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    Locked synchronizers:
  2>    - java.util.concurrent.ThreadPoolExecutor$Worker@76505305
  2> 
  2> "SUITE-TestLRUStatsCacheCloud-seed#[877B3D3D7C8CC18E]" ID=39 RUNNABLE
  2>    at 
java.management@11.0.16/sun.management.ThreadImpl.dumpThreads0(Native Method)
  2>    at 
java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:521)
  2>    at 
java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:509)
  2>    at 
app//com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:708)
  2>    at 
app//com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:59)
  2>    at 
app//com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:437)
  2>    - locked java.lang.Object@585b5dc3
  2>    at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:716)
  2>    at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:138)
  2>    at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:637)
  2> 
  2> "TEST-TestLRUStatsCacheCloud.testBasicStats-seed#[877B3D3D7C8CC18E]" ID=40 
WAITING on java.util.concurrent.FutureTask@5e95ac1e
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on java.util.concurrent.FutureTask@5e95ac1e
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.FutureTask.awaitDone(FutureTask.java:447)
  2>    at 
java.base@11.0.16/java.util.concurrent.FutureTask.get(FutureTask.java:190)
  2>    at 
java.base@11.0.16/java.util.concurrent.AbstractExecutorService.invokeAll(AbstractExecutorService.java:247)
  2>    at 
app//org.apache.solr.cloud.MiniSolrCloudCluster.<init>(MiniSolrCloudCluster.java:378)
  2>    at 
app//org.apache.solr.cloud.MiniSolrCloudCluster$Builder.build(MiniSolrCloudCluster.java:1303)
  2>    at 
app//org.apache.solr.cloud.MiniSolrCloudCluster$Builder.configure(MiniSolrCloudCluster.java:1275)
  2>    at 
app//org.apache.solr.cloud.TestBaseStatsCacheCloud.setupCluster(TestBaseStatsCacheCloud.java:86)
  2>    at 
java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
  2>    at 
java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2>    at 
java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2>    at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2>    at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  2>    at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:980)
  2>    at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  2>    at 
app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2>    at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>    at 
app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  2>    at 
app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>    at 
app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  2>    at 
app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2>    at 
app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2>    at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>    at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>    at 
app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2>    at 
app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  2>    at 
app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  2>    at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  2>    at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  2>    at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  2>    at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  2>    at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>    at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>    at 
app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>    at 
app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>    at 
app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2>    at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>    at 
app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>    at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>    at 
app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  2>    at 
app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>    at 
app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>    at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>    at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>    at 
app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2>    at 
app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>    at 
app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2>    at 
app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2>    at 
app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  2>    at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>    at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>    at 
app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2>    at 
app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  2>    at 
app//com.carrotsearch.randomizedtesting.ThreadLeakControl$$Lambda$186/0x000000084021ac40.run(Unknown
 Source)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "httpShardExecutor-13-thread-1" ID=41 RUNNABLE (in native code)
  2>    at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>    - locked sun.nio.ch.Util$2@260b6165
  2>    - locked sun.nio.ch.EPollSelectorImpl@3517b6d5
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$396/0x00000008404c8440.run(Unknown
 Source)
  2>    at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>    at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$261/0x0000000840317c40.run(Unknown
 Source)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    Locked synchronizers:
  2>    - java.util.concurrent.ThreadPoolExecutor$Worker@60b0ed4c
  2> 
  2> "httpShardExecutor-13-thread-2" ID=42 RUNNABLE (in native code)
  2>    at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>    - locked sun.nio.ch.Util$2@372050f1
  2>    - locked sun.nio.ch.EPollSelectorImpl@44a6ff57
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$396/0x00000008404c8440.run(Unknown
 Source)
  2>    at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>    at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$261/0x0000000840317c40.run(Unknown
 Source)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    Locked synchronizers:
  2>    - java.util.concurrent.ThreadPoolExecutor$Worker@1371ca51
  2> 
  2> "Connection evictor" ID=43 TIMED_WAITING
  2>    at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2>    at 
app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Connection evictor" ID=44 TIMED_WAITING
  2>    at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2>    at 
app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "h2sc-15-thread-1" ID=45 RUNNABLE (in native code)
  2>    at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>    - locked sun.nio.ch.Util$2@39dbbeef
  2>    - locked sun.nio.ch.EPollSelectorImpl@3172061f
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$396/0x00000008404c8440.run(Unknown
 Source)
  2>    at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>    at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$261/0x0000000840317c40.run(Unknown
 Source)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    Locked synchronizers:
  2>    - java.util.concurrent.ThreadPoolExecutor$Worker@32a3e50d
  2> 
  2> "h2sc-15-thread-2" ID=46 RUNNABLE (in native code)
  2>    at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>    - locked sun.nio.ch.Util$2@4e07f18
  2>    - locked sun.nio.ch.EPollSelectorImpl@77cf6494
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$396/0x00000008404c8440.run(Unknown
 Source)
  2>    at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>    at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$261/0x0000000840317c40.run(Unknown
 Source)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    Locked synchronizers:
  2>    - java.util.concurrent.ThreadPoolExecutor$Worker@7b9d58f5
  2> 
  2> "CloserThread" ID=48 WAITING on java.lang.Object@13078044
  2>    at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2>    - waiting on java.lang.Object@13078044
  2>    at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
  2>    at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2557)
  2> 
  2> "ForkJoinPool.commonPool-worker-3" ID=50 WAITING on 
java.util.concurrent.ForkJoinPool@32e42ec5
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on java.util.concurrent.ForkJoinPool@32e42ec5
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1628)
  2>    at 
java.base@11.0.16/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
  2> 
  2> "searcherExecutor-20-thread-1" ID=51 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35fb8f3e
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35fb8f3e
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "ZkTestServer Run Thread" ID=52 WAITING on 
org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread@391ca6d1
  2>    at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2>    - waiting on 
org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread@391ca6d1
  2>    at java.base@11.0.16/java.lang.Thread.join(Thread.java:1300)
  2>    at java.base@11.0.16/java.lang.Thread.join(Thread.java:1375)
  2>    at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:915)
  2>    at 
app//org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:337)
  2>    at app//org.apache.solr.cloud.ZkTestServer$1.run(ZkTestServer.java:581)
  2> 
  2> "NIOServerCxnFactory.SelectorThread-0" ID=54 RUNNABLE (in native code)
  2>    at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>    - locked sun.nio.ch.Util$2@16b60416
  2>    - locked sun.nio.ch.EPollSelectorImpl@5378203
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>    at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.select(NIOServerCnxnFactory.java:403)
  2>    at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.run(NIOServerCnxnFactory.java:368)
  2> 
  2> "NIOServerCxnFactory.SelectorThread-1" ID=55 RUNNABLE (in native code)
  2>    at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>    - locked sun.nio.ch.Util$2@6c5c6fe4
  2>    - locked sun.nio.ch.EPollSelectorImpl@30aa6690
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>    at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.select(NIOServerCnxnFactory.java:403)
  2>    at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.run(NIOServerCnxnFactory.java:368)
  2> 
  2> "NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:0" ID=56 RUNNABLE (in 
native code)
  2>    at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>    - locked sun.nio.ch.Util$2@161a5227
  2>    - locked sun.nio.ch.EPollSelectorImpl@2f04fd7d
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>    at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread.select(NIOServerCnxnFactory.java:205)
  2>    at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread.run(NIOServerCnxnFactory.java:181)
  2> 
  2> "ConnnectionExpirer" ID=53 TIMED_WAITING
  2>    at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2>    at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory$ConnectionExpirerThread.run(NIOServerCnxnFactory.java:554)
  2> 
  2> "SessionTracker" ID=57 TIMED_WAITING
  2>    at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2>    at 
app//org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:163)
  2> 
  2> "SyncThread:0" ID=58 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@20a88436
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@20a88436
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>    at 
app//org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:170)
  2> 
  2> "ProcessThread(sid:0 cport:41155):" ID=59 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@43d5f855
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@43d5f855
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>    at 
app//org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:142)
  2> 
  2> "RequestThrottler" ID=60 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@71a0162
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@71a0162
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>    at 
app//org.apache.zookeeper.server.RequestThrottler.run(RequestThrottler.java:147)
  2> 
  2> "NIOWorkerThread-1" ID=61 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f7debe9
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f7debe9
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> 
"TEST-TestLRUStatsCacheCloud.testBasicStats-seed#[877B3D3D7C8CC18E]-SendThread(127.0.0.1:41155)"
 ID=62 RUNNABLE (in native code)
  2>    at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>    - locked sun.nio.ch.Util$2@6b905704
  2>    - locked sun.nio.ch.EPollSelectorImpl@7eb1d73
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
  2>    at 
app//org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:332)
  2>    at 
app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289)
  2> 
  2> 
"TEST-TestLRUStatsCacheCloud.testBasicStats-seed#[877B3D3D7C8CC18E]-EventThread"
 ID=63 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@50ad1dfd
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@50ad1dfd
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>    at 
app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:549)
  2> 
  2> "NIOWorkerThread-2" ID=64 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f7debe9
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f7debe9
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "NIOWorkerThread-3" ID=65 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f7debe9
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f7debe9
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "zkConnectionManagerCallback-25-thread-1" ID=66 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@306f96a2
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@306f96a2
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> 
"TEST-TestLRUStatsCacheCloud.testBasicStats-seed#[877B3D3D7C8CC18E]-SendThread(127.0.0.1:41155)"
 ID=67 RUNNABLE (in native code)
  2>    at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>    - locked sun.nio.ch.Util$2@59aedfdc
  2>    - locked sun.nio.ch.EPollSelectorImpl@6f1511f
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
  2>    at 
app//org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:332)
  2>    at 
app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289)
  2> 
  2> 
"TEST-TestLRUStatsCacheCloud.testBasicStats-seed#[877B3D3D7C8CC18E]-EventThread"
 ID=68 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8be6c9
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e8be6c9
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>    at 
app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:549)
  2> 
  2> "zkConnectionManagerCallback-27-thread-1" ID=69 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@729dbd3c
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@729dbd3c
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "jetty-launcher-30-thread-1" ID=73 WAITING on 
java.util.concurrent.CountDownLatch$Sync@d2f1e43
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on java.util.concurrent.CountDownLatch$Sync@d2f1e43
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
  2>    at 
java.base@11.0.16/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
  2>    at 
app//org.apache.solr.servlet.CoreContainerProvider$ContextInitializationKey.waitForReadyService(CoreContainerProvider.java:522)
  2>    at 
app//org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:561)
  2>    at 
app//org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
  2>    at 
app//org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133)
  2>    at 
app//org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774)
  2>    at 
app//org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472)
  2>    at 
app//org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992)
  2>    at 
app//org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480)
  2>    at 
app//org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427)
  2>    - locked org.apache.solr.embedded.JettySolrRunner$1@5de2cc8c
  2>    at 
app//org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253)
  2>    at 
app//org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94)
  2>    at 
app//org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616)
  2>    at 
app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554)
  2>    - locked org.apache.solr.embedded.JettySolrRunner@51170786
  2>    at 
app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525)
  2>    at 
app//org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:567)
  2>    at 
app//org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:373)
  2>    at 
app//org.apache.solr.cloud.MiniSolrCloudCluster$$Lambda$703/0x00000008406d5440.call(Unknown
 Source)
  2>    at 
java.base@11.0.16/java.util.concurrent.FutureTask.run(FutureTask.java:264)
  2>    at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>    at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$261/0x0000000840317c40.run(Unknown
 Source)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    Locked synchronizers:
  2>    - java.util.concurrent.locks.ReentrantLock$NonfairSync@30cb1ec3
  2>    - java.util.concurrent.ThreadPoolExecutor$Worker@180f03d3
  2>    - java.util.concurrent.locks.ReentrantLock$NonfairSync@2fa031bb
  2>    - java.util.concurrent.locks.ReentrantLock$NonfairSync@64ed1e6f
  2> 
  2> "qtp1324057310-77" ID=77 RUNNABLE (in native code)
  2>    at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>    - locked sun.nio.ch.Util$2@657a2423
  2>    - locked sun.nio.ch.EPollSelectorImpl@4b79ecde
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$396/0x00000008404c8440.run(Unknown
 Source)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:936)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1080)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1324057310-78" ID=78 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@47eb9db8
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - timed waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@47eb9db8
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>    at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1017)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1067)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1324057310-79-acceptor-0@75feb42a-ServerConnector@3ab31b79{HTTP/1.1, 
(http/1.1, h2c)}{127.0.0.1:38599}" ID=79 RUNNABLE (in native code)
  2>    at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native 
Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
  2>    at 
java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
  2>    at 
app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
  2>    at 
app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:936)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1080)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    Locked synchronizers:
  2>    - java.util.concurrent.locks.ReentrantLock$NonfairSync@32435e00
  2> 
  2> "qtp1324057310-80" ID=80 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@47eb9db8
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - timed waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@47eb9db8
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>    at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1017)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1067)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1324057310-81" ID=81 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@47eb9db8
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - timed waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@47eb9db8
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>    at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1017)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1067)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1324057310-82" ID=82 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@47eb9db8
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - timed waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@47eb9db8
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>    at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1017)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1067)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp119531633-76" ID=76 RUNNABLE (in native code)
  2>    at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>    - locked sun.nio.ch.Util$2@6ac2f57
  2>    - locked sun.nio.ch.EPollSelectorImpl@2372c40
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$396/0x00000008404c8440.run(Unknown
 Source)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:936)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1080)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp119531633-84-acceptor-0@bf65271-ServerConnector@2cf369e5{HTTP/1.1, 
(http/1.1, h2c)}{127.0.0.1:34605}" ID=84 RUNNABLE (in native code)
  2>    at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native 
Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
  2>    at 
java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
  2>    at 
app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
  2>    at 
app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:936)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1080)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    Locked synchronizers:
  2>    - java.util.concurrent.locks.ReentrantLock$NonfairSync@19aa67e3
  2> 
  2> "qtp1324057310-83" ID=83 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@47eb9db8
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - timed waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@47eb9db8
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>    at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1017)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1067)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp119531633-85" ID=85 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@31b18988
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - timed waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@31b18988
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>    at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1017)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1067)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1324057310-86" ID=86 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@47eb9db8
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - timed waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@47eb9db8
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>    at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1017)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1067)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp119531633-87" ID=87 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@31b18988
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - timed waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@31b18988
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>    at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1017)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1067)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp119531633-88" ID=88 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@31b18988
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - timed waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@31b18988
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>    at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1017)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1067)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp119531633-89" ID=89 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@31b18988
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - timed waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@31b18988
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>    at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1017)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1067)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp119531633-90" ID=90 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@31b18988
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - timed waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@31b18988
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>    at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1017)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1067)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp119531633-91" ID=91 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@31b18988
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - timed waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@31b18988
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>    at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1017)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1067)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Session-HouseKeeper-7c51060f-1" ID=92 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5600de7e
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - timed waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5600de7e
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>    at 
java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2>    at 
java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Session-Hou

[...truncated too long message...]

 at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1017)
  2>         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1067)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   41) Thread[id=100, name=httpShardExecutor-44-thread-1, state=RUNNABLE, 
group=TGRP-TestLRUStatsCacheCloud]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$396/0x00000008404c8440.run(Unknown
 Source)
  2>         at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$261/0x0000000840317c40.run(Unknown
 Source)
  2>         at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   42) Thread[id=111, name=h2sc-48-thread-2, state=RUNNABLE, 
group=TGRP-TestLRUStatsCacheCloud]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$396/0x00000008404c8440.run(Unknown
 Source)
  2>         at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$261/0x0000000840317c40.run(Unknown
 Source)
  2>         at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   43) Thread[id=67, 
name=TEST-TestLRUStatsCacheCloud.testBasicStats-seed#[877B3D3D7C8CC18E]-SendThread(127.0.0.1:41155),
 state=RUNNABLE, group=TGRP-TestLRUStatsCacheCloud]
  2>         at java.base@11.0.16/sun.nio.ch.IOUtil.drain(Native Method)
  2>         at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.clearInterrupt(EPollSelectorImpl.java:266)
  2>         at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.processEvents(EPollSelectorImpl.java:203)
  2>         at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:137)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
  2>         at 
app//org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:332)
  2>         at 
app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289)
   >     java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   >         at __randomizedtesting.SeedInfo.seed([877B3D3D7C8CC18E]:0)
  2> NOTE: test params are: codec=Asserting(Lucene95): {}, docValues:{}, 
maxPointsInLeafNode=487, maxMBSortInHeap=7.366319819417228, 
sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=pa-Guru-IN, 
timezone=America/Maceio
  2> NOTE: Linux 5.4.0-100-generic amd64/Oracle Corporation 11.0.16 
(64-bit)/cpus=1,threads=51,free=206118072,total=337117184
  2> NOTE: All tests run in this JVM: [ConfigSetApiLockingTest, 
TestLRUStatsCacheCloud]
  2> NOTE: reproduce with: gradlew test --tests TestLRUStatsCacheCloud 
-Dtests.seed=877B3D3D7C8CC18E -Dtests.locale=pa-Guru-IN 
-Dtests.timezone=America/Maceio -Dtests.asserts=true -Dtests.file.encoding=UTF-8

:solr:core:test (FAILURE): 4740 test(s), 3 failure(s), 242 skipped

4740 tests completed, 3 failed, 242 skipped

> Task :solr:core:test FAILED
> Task :solr:core:wipeTaskTemp

ERROR: The following test(s) have failed:
  - org.apache.solr.cloud.OverseerTest.testShardLeaderChange (:solr:core)
    Test output: 
/tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.cloud.OverseerTest.txt
    Reproduce with: gradlew :solr:core:test --tests 
"org.apache.solr.cloud.OverseerTest.testShardLeaderChange" -Ptests.jvms=96 
"-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC 
-XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" 
-Ptests.seed=877B3D3D7C8CC18E -Ptests.file.encoding=UTF-8

  - org.apache.solr.cloud.TestLRUStatsCacheCloud.testBasicStats (:solr:core)
    Test output: 
/tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.cloud.TestLRUStatsCacheCloud.txt
    Reproduce with: gradlew :solr:core:test --tests 
"org.apache.solr.cloud.TestLRUStatsCacheCloud.testBasicStats" -Ptests.jvms=96 
"-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC 
-XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" 
-Ptests.seed=877B3D3D7C8CC18E -Ptests.file.encoding=UTF-8

  - org.apache.solr.cloud.TestLRUStatsCacheCloud.classMethod (:solr:core)
    Test output: 
/tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.cloud.TestLRUStatsCacheCloud.txt
    Reproduce with: gradlew :solr:core:test --tests 
"org.apache.solr.cloud.TestLRUStatsCacheCloud.classMethod" -Ptests.jvms=96 
"-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC 
-XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" 
-Ptests.seed=877B3D3D7C8CC18E -Ptests.file.encoding=UTF-8


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command 
> '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished 
> with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:core:test'.
> There were failing tests. See the results at: 
> file:///tmp/src/solr/solr/core/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 2h 3m 59s
625 actionable tasks: 609 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 2h4m24.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' 
'**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No 
such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 
23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscr...@solr.apache.org
For additional commands, e-mail: builds-h...@solr.apache.org