[JENKINS] Solr » Solr-Check-9.x - Build # 4528 - Still Failing!

[Apache Jenkins Server]

Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4528/

2 tests failed.
FAILED:  org.apache.solr.cloud.RollingRestartTest.test

Error Message:
java.lang.Exception: Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([F2B29286DDDD922A]:0)


FAILED:  org.apache.solr.cloud.RollingRestartTest.classMethod

Error Message:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([F2B29286DDDD922A]:0)




Build Log:
[...truncated 1220 lines...]
# ERROR: Error from server at 
http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2:
 org.apache.solr.cluster.placement.PlacementException: Not enough eligible 
nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME
ok 40 Random placement plugin using ENV in 22614ms
ok 41 post solr log into solr in 9024ms
ok 42 start solr with ssl in 19807ms
ok 43 start solr with ssl and auth in 22174ms
ok 44 SOLR-11740 check 'solr stop' connection in 28384ms
ok 45 status detects locally running solr in 14418ms
ok 46 status does not expose cli parameters to end user in 211ms
ok 47 listing out files in 2188ms
ok 48 copying files around in 6579ms
ok 49 upconfig in 2621ms

> Task :solr:packaging:integrationTests FAILED

> Task :solr:core:test

org.apache.solr.cloud.RollingRestartTest > test FAILED
    java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([F2B29286DDDD922A]:0)

org.apache.solr.cloud.RollingRestartTest > classMethod FAILED
    java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([F2B29286DDDD922A]:0)

org.apache.solr.cloud.RollingRestartTest > test suite's output saved to 
/tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.cloud.RollingRestartTest.txt,
 copied below:
  2> 11112 INFO  (SUITE-RollingRestartTest-seed#[F2B29286DDDD922A]-worker) [] 
o.a.s.SolrTestCase Setting 'solr.default.confdir' system property to 
test-framework derived value of 
'/tmp/src/solr/solr/server/solr/configsets/_default/conf'
  2> 11114 INFO  (SUITE-RollingRestartTest-seed#[F2B29286DDDD922A]-worker) [] 
o.a.s.u.ErrorLogMuter Closing ErrorLogMuter-regex-2 after mutting 0 log messages
  2> 11114 INFO  (SUITE-RollingRestartTest-seed#[F2B29286DDDD922A]-worker) [] 
o.a.s.u.ErrorLogMuter Creating ErrorLogMuter-regex-3 for ERROR logs matching 
regex: ignore_exception
  2> 11118 INFO  (SUITE-RollingRestartTest-seed#[F2B29286DDDD922A]-worker) [] 
o.a.s.SolrTestCaseJ4 Created dataDir: 
/tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.cloud.RollingRestartTest_F2B29286DDDD922A-001/data-dir-2-001
  2> 11119 WARN  (SUITE-RollingRestartTest-seed#[F2B29286DDDD922A]-worker) [] 
o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
  2> 11119 INFO  (SUITE-RollingRestartTest-seed#[F2B29286DDDD922A]-worker) [] 
o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) 
w/NUMERIC_DOCVALUES_SYSPROP=false
  2> 11121 INFO  (SUITE-RollingRestartTest-seed#[F2B29286DDDD922A]-worker) [] 
o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: 
@org.apache.solr.util.RandomizeSSL(reason="", value=0.0/0.0, ssl=0.0/0.0, 
clientAuth=0.0/0.0)
  2> 11136 INFO  (SUITE-RollingRestartTest-seed#[F2B29286DDDD922A]-worker) [] 
o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: 
test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
  2> 11137 INFO  (SUITE-RollingRestartTest-seed#[F2B29286DDDD922A]-worker) [] 
o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
  2> 11151 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.u.ErrorLogMuter Closing ErrorLogMuter-regex-3 after mutting 0 log messages
  2> 11152 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.u.ErrorLogMuter Creating ErrorLogMuter-regex-4 for ERROR logs matching 
regex: ignore_exception
  2> 11160 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
  2> 11184 INFO  (ZkTestServer Run Thread) [] o.a.s.c.ZkTestServer client port: 
0.0.0.0/0.0.0.0:0
  2> 11185 INFO  (ZkTestServer Run Thread) [] o.a.s.c.ZkTestServer Starting 
server
  2> 11280 WARN  (ZkTestServer Run Thread) [] o.a.z.s.ServerCnxnFactory 
maxCnxns is not configured, using default value 0.
  2> 11383 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.c.ZkTestServer start zk server on port: 46031
  2> 11431 WARN  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. 
ZkCredentialsInjector is not secure, it creates an empty list of credentials 
which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 11477 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to 
ZooKeeper
  2> 11516 INFO  (zkConnectionManagerCallback-22-thread-1) [] 
o.a.s.c.c.ConnectionManager zkClient has connected
  2> 11519 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 11519 WARN  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not 
secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 11546 WARN  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. 
ZkCredentialsInjector is not secure, it creates an empty list of credentials 
which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 11556 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to 
ZooKeeper
  2> 11565 INFO  (zkConnectionManagerCallback-24-thread-1) [] 
o.a.s.c.c.ConnectionManager zkClient has connected
  2> 11565 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 11566 WARN  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not 
secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 11586 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.c.ZkTestServer put 
/tmp/src/solr/solr/core/build/resources/test/solr/collection1/conf/solrconfig-tlog.xml
 to /configs/conf1/solrconfig.xml
  2> 11613 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.c.ZkTestServer put 
/tmp/src/solr/solr/core/build/resources/test/solr/collection1/conf/schema.xml 
to /configs/conf1/schema.xml
  2> 11621 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.c.ZkTestServer put 
/tmp/src/solr/solr/core/build/resources/test/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
 to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
  2> 11629 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.c.ZkTestServer put 
/tmp/src/solr/solr/core/build/resources/test/solr/collection1/conf/stopwords.txt
 to /configs/conf1/stopwords.txt
  2> 11635 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.c.ZkTestServer put 
/tmp/src/solr/solr/core/build/resources/test/solr/collection1/conf/protwords.txt
 to /configs/conf1/protwords.txt
  2> 11643 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.c.ZkTestServer put 
/tmp/src/solr/solr/core/build/resources/test/solr/collection1/conf/currency.xml 
to /configs/conf1/currency.xml
  2> 11652 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.c.ZkTestServer put 
/tmp/src/solr/solr/core/build/resources/test/solr/collection1/conf/enumsConfig.xml
 to /configs/conf1/enumsConfig.xml
  2> 11659 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.c.ZkTestServer put 
/tmp/src/solr/solr/core/build/resources/test/solr/collection1/conf/open-exchange-rates.json
 to /configs/conf1/open-exchange-rates.json
  2> 11671 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.c.ZkTestServer put 
/tmp/src/solr/solr/core/build/resources/test/solr/collection1/conf/mapping-ISOLatin1Accent.txt
 to /configs/conf1/mapping-ISOLatin1Accent.txt
  2> 11682 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.c.ZkTestServer put 
/tmp/src/solr/solr/core/build/resources/test/solr/collection1/conf/old_synonyms.txt
 to /configs/conf1/old_synonyms.txt
  2> 11695 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.c.ZkTestServer put 
/tmp/src/solr/solr/core/build/resources/test/solr/collection1/conf/synonyms.txt 
to /configs/conf1/synonyms.txt
  2> 11710 WARN  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. 
ZkCredentialsInjector is not secure, it creates an empty list of credentials 
which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 11740 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.c.c.ConnectionManager Waiting up to 45000ms for client to connect to 
ZooKeeper
  2> 11751 INFO  (zkConnectionManagerCallback-28-thread-1) [] 
o.a.s.c.c.ConnectionManager zkClient has connected
  2> 11751 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 11752 WARN  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not 
secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 11869 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly 
asked otherwise
  2> 12725 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
  2> 12725 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 2 ...
  2> 12731 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.e.j.s.Server jetty-10.0.14; built: 2023-02-22T23:12:32.272Z; git: 
976721d0f3e903a243584d47870ad2f2c1bf9e55; jvm 11.0.16+8
  2> 12774 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.e.j.s.s.DefaultSessionIdManager Session workerName=node0
  2> 12808 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@48a63e2c{/,file:///tmp/src/solr/solr/core/build/tmp/tests-cwd/,AVAILABLE}
  2> 12816 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.e.j.s.AbstractConnector Started ServerConnector@759c5476{ssl, (ssl, alpn, 
http/1.1, h2)}{127.0.0.1:40967}
  2> 12819 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.e.j.s.Server Started Server@127bcacd{STARTING}[10.0.14,sto=0] @12846ms
  2> 12829 ERROR (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
  2> 12830 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.s.CoreContainerProvider Using logger factory 
org.apache.logging.slf4j.Log4jLoggerFactory
  2> 12835 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.s.CoreContainerProvider  ___      _       Welcome to Apache Solr™ version 
9.3.0
  2> 12836 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.s.CoreContainerProvider / __| ___| |_ _   Starting in cloud mode on port 
null
  2> 12837 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_|  Install dir: /tmp/src/solr/solr
  2> 12837 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.s.CoreContainerProvider |___/\___/_|_|    Start time: 
2023-06-05T13:13:42.586682Z
  2> 12840 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.s.CoreContainerProvider Solr Home: 
/tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.cloud.RollingRestartTest_F2B29286DDDD922A-001/control-001
 (source: servlet config: solr.solr.home)
  2> 12841 WARN  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. 
ZkCredentialsInjector is not secure, it creates an empty list of credentials 
which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 12868 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to 
ZooKeeper
  2> 12871 INFO  (zkConnectionManagerCallback-30-thread-1) [] 
o.a.s.c.c.ConnectionManager zkClient has connected
  2> 12872 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 12872 WARN  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not 
secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 12983 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
  2> 12984 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.c.SolrXmlConfig Loading solr.xml from 
/tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.cloud.RollingRestartTest_F2B29286DDDD922A-001/control-001/solr.xml
  2> 14452 WARN  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.e.j.u.s.S.config Trusting all certificates configured for 
Client@1e11ba5[provider=null,keyStore=null,trustStore=null]
  2> 14456 WARN  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for 
Client@1e11ba5[provider=null,keyStore=null,trustStore=null]
  2> 14508 WARN  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.e.j.u.s.S.config Trusting all certificates configured for 
Client@881243c[provider=null,keyStore=null,trustStore=null]
  2> 14508 WARN  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for 
Client@881243c[provider=null,keyStore=null,trustStore=null]
  2> 14534 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46031/solr
  2> 14557 WARN  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. 
ZkCredentialsInjector is not secure, it creates an empty list of credentials 
which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 14596 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to 
ZooKeeper
  2> 14628 INFO  (zkConnectionManagerCallback-40-thread-1) [] 
o.a.s.c.c.ConnectionManager zkClient has connected
  2> 14629 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 14630 WARN  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not 
secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 14750 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) [] 
o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater 
with useDistributedStateUpdate=false. Solr will be using Overseer based cluster 
state updates.
  2> 14766 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) 
[n:127.0.0.1:40967_] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for 
client to connect to ZooKeeper
  2> 14777 INFO  (zkConnectionManagerCallback-42-thread-1) [] 
o.a.s.c.c.ConnectionManager zkClient has connected
  2> 14778 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) 
[n:127.0.0.1:40967_] o.a.s.c.c.ConnectionManager Client is connected to 
ZooKeeper
  2> 14960 WARN  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) 
[n:127.0.0.1:40967_] o.a.s.c.ZkController Contents of zookeeper /security.json 
are world-readable; consider setting up ACLs as described in 
https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
  2> 15021 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) 
[n:127.0.0.1:40967_] o.a.s.c.DistributedClusterStateUpdater Creating 
DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will 
be using Overseer based cluster state updates.
  2> 15051 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) 
[n:127.0.0.1:40967_] o.a.s.c.OverseerElectionContext I am going to be the 
leader 127.0.0.1:40967_
  2> 15061 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) 
[n:127.0.0.1:40967_] o.a.s.c.Overseer Overseer 
(id=72057618058444805-127.0.0.1:40967_-n_0000000000) starting
  2> 15222 INFO  
(OverseerStateUpdate-72057618058444805-127.0.0.1:40967_-n_0000000000) 
[n:127.0.0.1:40967_] o.a.s.c.Overseer Starting to work on the main queue : 
127.0.0.1:40967_
  2> 15223 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) 
[n:127.0.0.1:40967_] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:40967_
  2> 15231 INFO  (zkCallback-41-thread-1) [] o.a.s.c.c.ZkStateReader Updated 
live nodes from ZooKeeper... (0) -> (1)
  2> 15262 WARN  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) 
[n:127.0.0.1:40967_] o.a.s.c.CoreContainer Not all security plugins configured! 
 authentication=disabled authorization=disabled.  Solr is only as secure as you 
make it. Consider configuring authentication/authorization before exposing Solr 
to users internal or external.  See https://s.apache.org/solrsecurity for more 
info
  2> 15671 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) 
[n:127.0.0.1:40967_] o.a.s.c.CorePropertiesLocator Found 0 core definitions 
underneath 
/tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.cloud.RollingRestartTest_F2B29286DDDD922A-001/control-001/cores
  2> 15827 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) 
[n:127.0.0.1:40967_] o.a.s.j.SolrRequestAuthorizer Creating a new 
SolrRequestAuthorizer
  2> 15921 INFO  (TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]) 
[n:127.0.0.1:40967_] o.a.s.e.JettySolrRunner Jetty properties: {hostContext=/, 
solr.data.dir=/tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.cloud.RollingRestartTest_F2B29286DDDD922A-001/tempDir-001/control/data,
 hostPort=40967, 
coreRootDirectory=/tmp/src/solr/solr/core/build/tmp/tests-cwd/../../../../../../../../tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.cloud.RollingRestartTest_F2B29286DDDD922A-001/control-001/cores,
 replicaType=NRT}
  2> Sun 06, 2023 1:13:40 HENGIHENGI 
com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
  2> WARNING: Suite execution timed out: 
org.apache.solr.cloud.RollingRestartTest
  2> ==== jstack at approximately timeout time ====
  2> "Test worker" ID=1 WAITING on 
com.carrotsearch.randomizedtesting.RandomizedRunner$2@75b995d1
  2>    at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2>    - waiting on 
com.carrotsearch.randomizedtesting.RandomizedRunner$2@75b995d1
  2>    at java.base@11.0.16/java.lang.Thread.join(Thread.java:1300)
  2>    at java.base@11.0.16/java.lang.Thread.join(Thread.java:1375)
  2>    at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:647)
  2>    at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:504)
  2>    at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:108)
  2>    at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:57)
  2>    at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:39)
  2>    at 
org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
  2>    at 
org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:52)
  2>    at 
java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
  2>    at 
java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2>    at 
java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2>    at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2>    at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
  2>    at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
  2>    at 
org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
  2>    at 
org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
  2>    at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
  2>    at 
org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176)
  2>    at 
org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
  2>    at 
org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
  2>    at 
org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
  2>    at 
org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
  2>    at 
org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
  2>    at 
org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
  2>    at 
app//worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
  2>    at 
app//worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
  2> 
  2> "Reference Handler" ID=2 RUNNABLE
  2>    at 
java.base@11.0.16/java.lang.ref.Reference.waitForReferencePendingList(Native 
Method)
  2>    at 
java.base@11.0.16/java.lang.ref.Reference.processPendingReferences(Reference.java:241)
  2>    at 
java.base@11.0.16/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:213)
  2> 
  2> "Finalizer" ID=3 WAITING on java.lang.ref.ReferenceQueue$Lock@48d90f16
  2>    at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2>    - waiting on java.lang.ref.ReferenceQueue$Lock@48d90f16
  2>    at 
java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2>    at 
java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:176)
  2>    at 
java.base@11.0.16/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:170)
  2> 
  2> "Signal Dispatcher" ID=4 RUNNABLE
  2> 
  2> "Common-Cleaner" ID=9 TIMED_WAITING on 
java.lang.ref.ReferenceQueue$Lock@71f70cc1
  2>    at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2>    - timed waiting on java.lang.ref.ReferenceQueue$Lock@71f70cc1
  2>    at 
java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2>    at 
java.base@11.0.16/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:148)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    at 
java.base@11.0.16/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:161)
  2> 
  2> "/127.0.0.1:55386 to /127.0.0.1:41783 workers" ID=11 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@822f29d
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@822f29d
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
java.base@11.0.16/java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:367)
  2>    at 
org.gradle.api.internal.tasks.testing.worker.TestWorker.submitToRun(TestWorker.java:207)
  2>    - locked 
org.gradle.api.internal.tasks.testing.worker.TestWorker@5f44e674
  2>    at 
org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:169)
  2>    at 
java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
  2>    at 
java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2>    at 
java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2>    at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2>    at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
  2>    at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
  2>    at 
org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
  2>    at 
org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
  2>    at 
org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
  2>    at 
org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2>    at 
org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    Locked synchronizers:
  2>    - java.util.concurrent.ThreadPoolExecutor$Worker@9f116cc
  2> 
  2> "/127.0.0.1:55386 to /127.0.0.1:41783 workers Thread 2" ID=12 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@803ed30
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@803ed30
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
org.gradle.internal.remote.internal.hub.queue.EndPointQueue.take(EndPointQueue.java:49)
  2>    at 
org.gradle.internal.remote.internal.hub.MessageHub$ConnectionDispatch.run(MessageHub.java:322)
  2>    at 
org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2>    at 
org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    Locked synchronizers:
  2>    - java.util.concurrent.ThreadPoolExecutor$Worker@5656be13
  2> 
  2> "/127.0.0.1:55386 to /127.0.0.1:41783 workers Thread 3" ID=13 RUNNABLE (in 
native code)
  2>    at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>    - locked sun.nio.ch.Util$2@415b69f5
  2>    - locked sun.nio.ch.EPollSelectorImpl@39d450ec
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>    at 
org.gradle.internal.remote.internal.inet.SocketConnection$SocketInputStream.read(SocketConnection.java:185)
  2>    at com.esotericsoftware.kryo.io.Input.fill(Input.java:146)
  2>    at com.esotericsoftware.kryo.io.Input.require(Input.java:178)
  2>    at com.esotericsoftware.kryo.io.Input.readByte(Input.java:295)
  2>    at 
org.gradle.internal.serialize.kryo.KryoBackedDecoder.readByte(KryoBackedDecoder.java:88)
  2>    at 
org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:64)
  2>    at 
org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:52)
  2>    at 
org.gradle.internal.remote.internal.inet.SocketConnection.receive(SocketConnection.java:81)
  2>    at 
org.gradle.internal.remote.internal.hub.MessageHub$ConnectionReceive.run(MessageHub.java:270)
  2>    at 
org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2>    at 
org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    Locked synchronizers:
  2>    - java.util.concurrent.ThreadPoolExecutor$Worker@77888435
  2> 
  2> "ForkJoinPool.commonPool-worker-3" ID=25 WAITING on 
java.util.concurrent.ForkJoinPool@267deb51
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on java.util.concurrent.ForkJoinPool@267deb51
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1628)
  2>    at 
java.base@11.0.16/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
  2> 
  2> "SUITE-RollingRestartTest-seed#[F2B29286DDDD922A]" ID=33 RUNNABLE
  2>    at 
java.management@11.0.16/sun.management.ThreadImpl.dumpThreads0(Native Method)
  2>    at 
java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:521)
  2>    at 
java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:509)
  2>    at 
app//com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:708)
  2>    at 
app//com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:59)
  2>    at 
app//com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:437)
  2>    - locked java.lang.Object@3180f8c3
  2>    at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:716)
  2>    at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:138)
  2>    at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:637)
  2> 
  2> "TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]" ID=34 WAITING on 
java.util.concurrent.CountDownLatch$Sync@17fdce34
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on java.util.concurrent.CountDownLatch$Sync@17fdce34
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
  2>    at 
java.base@11.0.16/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
  2>    at 
app//org.apache.solr.servlet.CoreContainerProvider$ContextInitializationKey.waitForReadyService(CoreContainerProvider.java:510)
  2>    at 
app//org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:549)
  2>    at 
app//org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
  2>    at 
app//org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133)
  2>    at 
app//org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774)
  2>    at 
app//org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472)
  2>    at 
app//org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992)
  2>    at 
app//org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480)
  2>    at 
app//org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427)
  2>    - locked org.apache.solr.embedded.JettySolrRunner$1@2b0e2bc7
  2>    at 
app//org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253)
  2>    at 
app//org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94)
  2>    at 
app//org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616)
  2>    at 
app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554)
  2>    - locked org.apache.solr.embedded.JettySolrRunner@30465d7a
  2>    at 
app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525)
  2>    at 
app//org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:347)
  2>    at 
app//org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1161)
  2>    at 
app//org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1134)
  2>    at 
app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2>    at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>    at 
app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  2>    at 
app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>    at 
app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  2>    at 
app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2>    at 
app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2>    at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>    at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>    at 
app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2>    at 
app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  2>    at 
app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  2>    at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  2>    at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  2>    at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  2>    at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  2>    at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>    at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>    at 
app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>    at 
app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>    at 
app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2>    at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>    at 
app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>    at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>    at 
app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  2>    at 
app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>    at 
app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>    at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>    at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>    at 
app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2>    at 
app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>    at 
app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2>    at 
app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2>    at 
app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  2>    at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>    at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>    at 
app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2>    at 
app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  2>    at 
app//com.carrotsearch.randomizedtesting.ThreadLeakControl$$Lambda$186/0x000000084021ac40.run(Unknown
 Source)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    Locked synchronizers:
  2>    - java.util.concurrent.locks.ReentrantLock$NonfairSync@79280f40
  2>    - java.util.concurrent.locks.ReentrantLock$NonfairSync@4f58b646
  2>    - java.util.concurrent.locks.ReentrantLock$NonfairSync@15d2ec79
  2> 
  2> "ZkTestServer Run Thread" ID=35 WAITING on 
org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread@2e168184
  2>    at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2>    - waiting on 
org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread@2e168184
  2>    at java.base@11.0.16/java.lang.Thread.join(Thread.java:1300)
  2>    at java.base@11.0.16/java.lang.Thread.join(Thread.java:1375)
  2>    at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:915)
  2>    at 
app//org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:337)
  2>    at app//org.apache.solr.cloud.ZkTestServer$1.run(ZkTestServer.java:581)
  2> 
  2> "NIOServerCxnFactory.SelectorThread-0" ID=37 RUNNABLE (in native code)
  2>    at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>    - locked sun.nio.ch.Util$2@24f6e886
  2>    - locked sun.nio.ch.EPollSelectorImpl@26c57945
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>    at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.select(NIOServerCnxnFactory.java:403)
  2>    at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.run(NIOServerCnxnFactory.java:368)
  2> 
  2> "NIOServerCxnFactory.SelectorThread-1" ID=38 RUNNABLE (in native code)
  2>    at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>    - locked sun.nio.ch.Util$2@5f308fc7
  2>    - locked sun.nio.ch.EPollSelectorImpl@45b02f04
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>    at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.select(NIOServerCnxnFactory.java:403)
  2>    at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.run(NIOServerCnxnFactory.java:368)
  2> 
  2> "NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:0" ID=39 RUNNABLE (in 
native code)
  2>    at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>    - locked sun.nio.ch.Util$2@56914dec
  2>    - locked sun.nio.ch.EPollSelectorImpl@5b13092a
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>    at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread.select(NIOServerCnxnFactory.java:205)
  2>    at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread.run(NIOServerCnxnFactory.java:181)
  2> 
  2> "ConnnectionExpirer" ID=36 TIMED_WAITING
  2>    at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2>    at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory$ConnectionExpirerThread.run(NIOServerCnxnFactory.java:554)
  2> 
  2> "SessionTracker" ID=40 TIMED_WAITING
  2>    at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2>    at 
app//org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:163)
  2> 
  2> "SyncThread:0" ID=41 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6fb38
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6fb38
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>    at 
app//org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:170)
  2> 
  2> "ProcessThread(sid:0 cport:46031):" ID=42 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@440a3c33
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@440a3c33
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>    at 
app//org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:142)
  2> 
  2> "RequestThrottler" ID=43 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@18a8f773
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@18a8f773
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>    at 
app//org.apache.zookeeper.server.RequestThrottler.run(RequestThrottler.java:147)
  2> 
  2> "NIOWorkerThread-1" ID=44 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5b31e715
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5b31e715
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> 
"TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]-SendThread(127.0.0.1:46031)"
 ID=45 RUNNABLE (in native code)
  2>    at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>    - locked sun.nio.ch.Util$2@6cd90870
  2>    - locked sun.nio.ch.EPollSelectorImpl@79a74a7c
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
  2>    at 
app//org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:332)
  2>    at 
app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289)
  2> 
  2> "TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]-EventThread" ID=46 
WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@266b2954
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@266b2954
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>    at 
app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:549)
  2> 
  2> "NIOWorkerThread-2" ID=47 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5b31e715
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5b31e715
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "NIOWorkerThread-3" ID=48 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5b31e715
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5b31e715
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "zkConnectionManagerCallback-22-thread-1" ID=49 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@72b66bc3
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@72b66bc3
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> 
"TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]-SendThread(127.0.0.1:46031)"
 ID=50 RUNNABLE (in native code)
  2>    at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>    - locked sun.nio.ch.Util$2@5688676d
  2>    - locked sun.nio.ch.EPollSelectorImpl@15bf336a
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
  2>    at 
app//org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:332)
  2>    at 
app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289)
  2> 
  2> "TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]-EventThread" ID=51 
WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6ae18900
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6ae18900
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>    at 
app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:549)
  2> 
  2> "zkConnectionManagerCallback-24-thread-1" ID=52 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2b480fec
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2b480fec
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp152692393-57" ID=57 RUNNABLE (in native code)
  2>    at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>    - locked sun.nio.ch.Util$2@3eeea6e7
  2>    - locked sun.nio.ch.EPollSelectorImpl@52a6f633
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$328/0x0000000840424840.run(Unknown
 Source)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:936)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1080)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp152692393-58-acceptor-0@5457a875-ServerConnector@759c5476{ssl, (ssl, 
alpn, http/1.1, h2)}{127.0.0.1:40967}" ID=58 RUNNABLE (in native code)
  2>    at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native 
Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
  2>    at 
java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
  2>    at 
app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
  2>    at 
app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:936)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1080)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    Locked synchronizers:
  2>    - java.util.concurrent.locks.ReentrantLock$NonfairSync@34e9ccee
  2> 
  2> "qtp152692393-59" ID=59 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5f3592f0
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - timed waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5f3592f0
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>    at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1017)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1067)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp152692393-60" ID=60 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5f3592f0
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - timed waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5f3592f0
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>    at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1017)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1067)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp152692393-61" ID=61 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5f3592f0
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - timed waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5f3592f0
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>    at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1017)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1067)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp152692393-62" ID=62 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5f3592f0
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - timed waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5f3592f0
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>    at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1017)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1067)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp152692393-63" ID=63 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5f3592f0
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - timed waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5f3592f0
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>    at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1017)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1067)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp152692393-64" ID=64 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5f3592f0
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - timed waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5f3592f0
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>    at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1017)
  2>    at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1067)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Session-HouseKeeper-71741b44-1" ID=65 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1427ce6f
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - timed waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1427ce6f
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>    at 
java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2>    at 
java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "httpShardExecutor-35-thread-1" ID=69 RUNNABLE (in native code)
  2>    at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>    - locked sun.nio.ch.Util$2@71d5b5b6
  2>    - locked sun.nio.ch.EPollSelectorImpl@b2199de
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$328/0x0000000840424840.run(Unknown
 Source)
  2>    at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>    at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$329/0x0000000840424040.run(Unknown
 Source)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    Locked synchronizers:
  2>    - java.util.concurrent.ThreadPoolExecutor$Worker@6c9d0f85
  2> 
  2> "httpShardExecutor-35-thread-2" ID=70 RUNNABLE (in native code)
  2>    at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>    - locked sun.nio.ch.Util$2@78f03858
  2>    - locked sun.nio.ch.EPollSelectorImpl@528ec59f
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$328/0x0000000840424840.run(Unknown
 Source)
  2>    at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>    at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$329/0x0000000840424040.run(Unknown
 Source)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    Locked synchronizers:
  2>    - java.util.concurrent.ThreadPoolExecutor$Worker@37784dd6
  2> 
  2> "Connection evictor" ID=71 TIMED_WAITING
  2>    at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2>    at 
app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Connection evictor" ID=72 TIMED_WAITING
  2>    at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2>    at 
app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "h2sc-37-thread-1" ID=73 RUNNABLE (in native code)
  2>    at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>    - locked sun.nio.ch.Util$2@eaaa27d
  2>    - locked sun.nio.ch.EPollSelectorImpl@9c2351f
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$328/0x0000000840424840.run(Unknown
 Source)
  2>    at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>    at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$329/0x0000000840424040.run(Unknown
 Source)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    Locked synchronizers:
  2>    - java.util.concurrent.ThreadPoolExecutor$Worker@407ece3a
  2> 
  2> "h2sc-37-thread-2" ID=74 RUNNABLE (in native code)
  2>    at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>    - locked sun.nio.ch.Util$2@7089fe87
  2>    - locked sun.nio.ch.EPollSelectorImpl@548f6f34
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>    at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>    at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$328/0x0000000840424840.run(Unknown
 Source)
  2>    at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>    at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$329/0x0000000840424040.run(Unknown
 Source)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    Locked synchronizers:
  2>    - java.util.concurrent.ThreadPoolExecutor$Worker@2735355b
  2> 
  2> 
"TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]-SendThread(127.0.0.1:46031)"
 ID=78 RUNNABLE (in native code)
  2>    at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>    at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>    - locked sun.nio.ch.Util$2@fed8692
  2>    - locked sun.nio.ch.EPollSelectorImpl@179f20bd
  2>    at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
  2>    at 
app//org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:332)
  2>    at 
app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289)
  2> 
  2> "TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]-EventThread" ID=79 
WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@28dd990f
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@28dd990f
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>    at 
app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:549)
  2> 
  2> "zkConnectionManagerCallback-42-thread-1" ID=80 WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@784a5ef
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@784a5ef
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>    at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>    at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "OverseerStateUpdate-72057618058444805-127.0.0.1:40967_-n_0000000000" 
ID=81 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@643e7a9b
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - timed waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@643e7a9b
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>    at 
app//org.apache.solr.cloud.ZkDistributedQueue.peekElements(ZkDistributedQueue.java:490)
  2>    at 
app//org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:338)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> 
"OverseerCollectionConfigSetProcessor-72057618058444805-127.0.0.1:40967_-n_0000000000"
 ID=82 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@a4a3469
  2>    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>    - timed waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@a4a3469
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>    at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>    at 
app//org.apache.solr.cloud.ZkDistributedQueue.peekElements(ZkDistributedQueue.java:490)
  2>    at 
app//org.apache.solr.cloud.OverseerTaskQueue.peekTopN(OverseerTaskQueue.java:267)
  2>    at 
app//org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:272)
  2>    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "CloserThread" ID=84 WAITING on java.lang.Object@4e9bc1ce
  2>    at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2>    - waiting on java.lang.Object@4e9bc1ce
  2>    at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
  2>    at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2554)
  2> 
  2> ^^==============================================
  2> 
   >     java.lang.Exception: Test abandoned because suite timeout was reached.
   >         at __randomizedtesting.SeedInfo.seed([F2B29286DDDD922A]:0)
  2> NOTE: reproduce with: gradlew test --tests RollingRestartTest.test 
-Dtests.seed=F2B29286DDDD922A -Dtests.locale=to-TO -Dtests.timezone=Pacific/Yap 
-Dtests.asserts=true -Dtests.file.encoding=UTF-8
  2> Sun 06, 2023 1:13:40 HENGIHENGI 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> WARNING: Will linger awaiting termination of 36 leaked thread(s).
  2> Sun 06, 2023 1:13:41 HENGIHENGI 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> SEVERE: 36 threads leaked from SUITE scope at 
org.apache.solr.cloud.RollingRestartTest: 
  2>    1) Thread[id=84, name=CloserThread, state=WAITING, 
group=TGRP-RollingRestartTest]
  2>         at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2>         at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
  2>         at 
app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2554)
  2>    2) Thread[id=70, name=httpShardExecutor-35-thread-2, state=RUNNABLE, 
group=TGRP-RollingRestartTest]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$328/0x0000000840424840.run(Unknown
 Source)
  2>         at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$329/0x0000000840424040.run(Unknown
 Source)
  2>         at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    3) Thread[id=73, name=h2sc-37-thread-1, state=RUNNABLE, 
group=TGRP-RollingRestartTest]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$328/0x0000000840424840.run(Unknown
 Source)
  2>         at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$329/0x0000000840424040.run(Unknown
 Source)
  2>         at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    4) Thread[id=59, name=qtp152692393-59, state=TIMED_WAITING, 
group=TGRP-RollingRestartTest]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1017)
  2>         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1067)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    5) Thread[id=78, 
name=TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]-SendThread(127.0.0.1:46031),
 state=RUNNABLE, group=TGRP-RollingRestartTest]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
  2>         at 
app//org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:332)
  2>         at 
app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289)
  2>    6) Thread[id=64, name=qtp152692393-64, state=TIMED_WAITING, 
group=TGRP-RollingRestartTest]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1017)
  2>         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1067)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    7) Thread[id=35, name=ZkTestServer Run Thread, state=WAITING, 
group=TGRP-RollingRestartTest]
  2>         at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2>         at java.base@11.0.16/java.lang.Thread.join(Thread.java:1300)
  2>         at java.base@11.0.16/java.lang.Thread.join(Thread.java:1375)
  2>         at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:915)
  2>         at 
app//org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:337)
  2>         at 
app//org.apache.solr.cloud.ZkTestServer$1.run(ZkTestServer.java:581)
  2>    8) Thread[id=80, name=zkConnectionManagerCallback-42-thread-1, 
state=WAITING, group=TGRP-RollingRestartTest]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>         at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>         at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    9) Thread[id=38, name=NIOServerCxnFactory.SelectorThread-1, 
state=RUNNABLE, group=TGRP-RollingRestartTest]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.select(NIOServerCnxnFactory.java:403)
  2>         at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.run(NIOServerCnxnFactory.java:368)
  2>   10) Thread[id=60, name=qtp152692393-60, state=TIMED_WAITING, 
group=TGRP-RollingRestartTest]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at 
app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1017)
  2>         at 
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1067)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   11) Thread[id=74, name=h2sc-37-thread-2, state=RUNNABLE, 
group=TGRP-RollingRestartTest]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$328/0x0000000840424840.run(Unknown
 Source)
  2>         at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$329/0x0000000840424040.run(Unknown
 Source)
  2>         at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   12) Thread[id=39, 
name=NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:0, state=RUNNABLE, 
group=TGRP-RollingRestartTest]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread.select(NIOServerCnxnFactory.java:205)
  2>         at 
app//org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread.run(NIOServerCnxnFactory.java:181)
  2>   13) Thread[id=50, 
name=TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]-SendThread(127.0.0.1:46031),
 state=RUNNABLE, group=TGRP-RollingRestartTest]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
  2>         at 
app//org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:332)
  2>         at 
app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289)
  2>   14) Thread[id=57, name=qtp152692393-57, state=RUNNABLE, 
group=TGRP-RollingRestartTest]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.

[...truncated too long message...]

ecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$328/0x0000000840424840.run(Unknown
 Source)
  2>         at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$329/0x0000000840424040.run(Unknown
 Source)
  2>         at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    3) Thread[id=73, name=h2sc-37-thread-1, state=RUNNABLE, 
group=TGRP-RollingRestartTest]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$328/0x0000000840424840.run(Unknown
 Source)
  2>         at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$329/0x0000000840424040.run(Unknown
 Source)
  2>         at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    4) Thread[id=78, 
name=TEST-RollingRestartTest.test-seed#[F2B29286DDDD922A]-SendThread(127.0.0.1:46031),
 state=TIMED_WAITING, group=TGRP-RollingRestartTest]
  2>         at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2>         at 
app//org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1142)
  2>         at 
app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1207)
  2>    5) Thread[id=80, name=zkConnectionManagerCallback-42-thread-1, 
state=WAITING, group=TGRP-RollingRestartTest]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at 
java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at 
java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>         at 
java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>         at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    6) Thread[id=69, name=httpShardExecutor-35-thread-1, state=RUNNABLE, 
group=TGRP-RollingRestartTest]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at 
java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at 
java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at 
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at 
app//org.eclipse.jetty.io.ManagedSelector$$Lambda$328/0x0000000840424840.run(Unknown
 Source)
  2>         at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at 
app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$329/0x0000000840424040.run(Unknown
 Source)
  2>         at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at 
java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   >     java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   >         at __randomizedtesting.SeedInfo.seed([F2B29286DDDD922A]:0)
  2> NOTE: test params are: codec=CheapBastard, 
sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=to-TO, 
timezone=Pacific/Yap
  2> NOTE: Linux 5.4.0-100-generic amd64/Oracle Corporation 11.0.16 
(64-bit)/cpus=1,threads=10,free=251813112,total=327155712
  2> NOTE: All tests run in this JVM: [TestCharFilters, RollingRestartTest]
  2> NOTE: reproduce with: gradlew test --tests RollingRestartTest 
-Dtests.seed=F2B29286DDDD922A -Dtests.locale=to-TO -Dtests.timezone=Pacific/Yap 
-Dtests.asserts=true -Dtests.file.encoding=UTF-8

:solr:core:test (FAILURE): 4757 test(s), 2 failure(s), 243 skipped

4757 tests completed, 2 failed, 243 skipped

> Task :solr:core:test FAILED
> Task :solr:core:wipeTaskTemp

ERROR: The following test(s) have failed:
  - org.apache.solr.cloud.RollingRestartTest.test (:solr:core)
    Test output: 
/tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.cloud.RollingRestartTest.txt
    Reproduce with: gradlew :solr:core:test --tests 
"org.apache.solr.cloud.RollingRestartTest.test" -Ptests.jvms=96 
"-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC 
-XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" 
-Ptests.seed=F2B29286DDDD922A -Ptests.file.encoding=ISO-8859-1

  - org.apache.solr.cloud.RollingRestartTest.classMethod (:solr:core)
    Test output: 
/tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.cloud.RollingRestartTest.txt
    Reproduce with: gradlew :solr:core:test --tests 
"org.apache.solr.cloud.RollingRestartTest.classMethod" -Ptests.jvms=96 
"-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC 
-XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" 
-Ptests.seed=F2B29286DDDD922A -Ptests.file.encoding=ISO-8859-1


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command 
> '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished 
> with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:core:test'.
> There were failing tests. See the results at: 
> file:///tmp/src/solr/solr/core/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 2h 3m 53s
625 actionable tasks: 610 executed, 15 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 2h4m16.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' 
'**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No 
such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 
23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscr...@solr.apache.org
For additional commands, e-mail: builds-h...@solr.apache.org