Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1786/

2 tests failed.
FAILED:  
org.apache.solr.cloud.api.collections.HdfsCollectionsAPIDistributedZkTest.testCollectionsAPI

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([A23D00C6F401CC94]:0)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.api.collections.HdfsCollectionsAPIDistributedZkTest

Error Message:
Suite timeout exceeded (>= 3600000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 3600000 msec).
        at __randomizedtesting.SeedInfo.seed([A23D00C6F401CC94]:0)




Build Log:
[...truncated 14385 lines...]
   [junit4] Suite: 
org.apache.solr.cloud.api.collections.HdfsCollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.api.collections.HdfsCollectionsAPIDistributedZkTest_A23D00C6F401CC94-001/init-core-data-001
   [junit4]   2> 395384 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A23D00C6F401CC94]-worker) [    
] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) 
w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 395385 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A23D00C6F401CC94]-worker) [    
] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: 
@org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 395385 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A23D00C6F401CC94]-worker) [    
] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: 
test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 396158 WARN  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A23D00C6F401CC94]-worker) [    
] o.a.h.u.NativeCodeLoader Unable to load native-hadoop library for your 
platform... using builtin-java classes where applicable
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 397732 WARN  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A23D00C6F401CC94]-worker) [    
] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried 
hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 398052 WARN  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A23D00C6F401CC94]-worker) [    
] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 398096 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A23D00C6F401CC94]-worker) [    
] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: 
c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 398102 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A23D00C6F401CC94]-worker) [    
] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 398102 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A23D00C6F401CC94]-worker) [    
] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 398102 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A23D00C6F401CC94]-worker) [    
] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 398105 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A23D00C6F401CC94]-worker) [    
] o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@655c873d{static,/static,jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-3.2.0-tests.jar!/webapps/static,AVAILABLE}
   [junit4]   2> 398404 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A23D00C6F401CC94]-worker) [    
] o.e.j.s.h.ContextHandler Started 
o.e.j.w.WebAppContext@33b20724{hdfs,/,file:///x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/jetty-localhost-43104-hdfs-_-any-6596358922321797196.dir/webapp/,AVAILABLE}{/hdfs}
   [junit4]   2> 398405 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A23D00C6F401CC94]-worker) [    
] o.e.j.s.AbstractConnector Started 
ServerConnector@538519af{HTTP/1.1,[http/1.1]}{localhost:43104}
   [junit4]   2> 398405 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A23D00C6F401CC94]-worker) [    
] o.e.j.s.Server Started @398467ms
   [junit4]   2> 400089 WARN  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A23D00C6F401CC94]-worker) [    
] o.a.h.h.s.c.MetricsLoggerTask Metrics logging will not be async since the 
logger is not log4j
   [junit4]   2> 400518 WARN  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A23D00C6F401CC94]-worker) [    
] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 400527 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A23D00C6F401CC94]-worker) [    
] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: 
c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 400528 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A23D00C6F401CC94]-worker) [    
] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 400528 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A23D00C6F401CC94]-worker) [    
] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 400528 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A23D00C6F401CC94]-worker) [    
] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 400529 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A23D00C6F401CC94]-worker) [    
] o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@253f748e{static,/static,jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-3.2.0-tests.jar!/webapps/static,AVAILABLE}
   [junit4]   2> 400707 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A23D00C6F401CC94]-worker) [    
] o.e.j.s.h.ContextHandler Started 
o.e.j.w.WebAppContext@11130f4{datanode,/,file:///x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/jetty-localhost-37309-datanode-_-any-3185228825448904120.dir/webapp/,AVAILABLE}{/datanode}
   [junit4]   2> 400708 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A23D00C6F401CC94]-worker) [    
] o.e.j.s.AbstractConnector Started 
ServerConnector@7f166b0e{HTTP/1.1,[http/1.1]}{localhost:37309}
   [junit4]   2> 400708 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A23D00C6F401CC94]-worker) [    
] o.e.j.s.Server Started @400770ms
   [junit4]   2> 401443 WARN  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A23D00C6F401CC94]-worker) [    
] o.a.h.h.s.c.MetricsLoggerTask Metrics logging will not be async since the 
logger is not log4j
   [junit4]   2> 402890 INFO  (Block report processor) [    ] BlockStateChange 
BLOCK* processReport 0xf3db7a82292f541b: Processing first storage report for 
DS-f8e34ba7-3576-41ce-a2e8-3b11a62ec1ca from datanode 
9ea10e01-9eb0-4f73-be10-9edab98b1871
   [junit4]   2> 402894 INFO  (Block report processor) [    ] BlockStateChange 
BLOCK* processReport 0xf3db7a82292f541b: from storage 
DS-f8e34ba7-3576-41ce-a2e8-3b11a62ec1ca node 
DatanodeRegistration(127.0.0.1:43008, 
datanodeUuid=9ea10e01-9eb0-4f73-be10-9edab98b1871, infoPort=33172, 
infoSecurePort=0, ipcPort=37000, 
storageInfo=lv=-57;cid=testClusterID;nsid=1118631180;c=1552064342193), blocks: 
0, hasStaleStorage: true, processing time: 4 msecs, invalidatedBlocks: 0
   [junit4]   2> 402894 INFO  (Block report processor) [    ] BlockStateChange 
BLOCK* processReport 0xf3db7a82292f541b: Processing first storage report for 
DS-27393714-9b93-4b9b-bfc2-49d33ca5f4a3 from datanode 
9ea10e01-9eb0-4f73-be10-9edab98b1871
   [junit4]   2> 402894 INFO  (Block report processor) [    ] BlockStateChange 
BLOCK* processReport 0xf3db7a82292f541b: from storage 
DS-27393714-9b93-4b9b-bfc2-49d33ca5f4a3 node 
DatanodeRegistration(127.0.0.1:43008, 
datanodeUuid=9ea10e01-9eb0-4f73-be10-9edab98b1871, infoPort=33172, 
infoSecurePort=0, ipcPort=37000, 
storageInfo=lv=-57;cid=testClusterID;nsid=1118631180;c=1552064342193), blocks: 
0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0
   [junit4]   2> 403216 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testCreationAndDeletion-seed#[A23D00C6F401CC94])
 [    ] o.a.s.SolrTestCaseJ4 ###Starting testCreationAndDeletion
   [junit4]   2> 403218 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testCreationAndDeletion-seed#[A23D00C6F401CC94])
 [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.api.collections.HdfsCollectionsAPIDistributedZkTest_A23D00C6F401CC94-001/tempDir-002
   [junit4]   2> 403218 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testCreationAndDeletion-seed#[A23D00C6F401CC94])
 [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 403228 INFO  (ZkTestServer Run Thread) [    ] 
o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 403228 INFO  (ZkTestServer Run Thread) [    ] 
o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 403328 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testCreationAndDeletion-seed#[A23D00C6F401CC94])
 [    ] o.a.s.c.ZkTestServer start zk server on port:39876
   [junit4]   2> 403328 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testCreationAndDeletion-seed#[A23D00C6F401CC94])
 [    ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:39876
   [junit4]   2> 403328 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testCreationAndDeletion-seed#[A23D00C6F401CC94])
 [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 39876
   [junit4]   2> 403358 INFO  (zkConnectionManagerCallback-2294-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 403364 INFO  (zkConnectionManagerCallback-2296-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 403366 INFO  (zkConnectionManagerCallback-2298-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 403373 WARN  (jetty-launcher-2299-thread-2) [    ] 
o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 403374 INFO  (jetty-launcher-2299-thread-2) [    ] 
o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 403374 INFO  (jetty-launcher-2299-thread-2) [    ] 
o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 403374 INFO  (jetty-launcher-2299-thread-2) [    ] 
o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: 
c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 403374 WARN  (jetty-launcher-2299-thread-3) [    ] 
o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 403375 INFO  (jetty-launcher-2299-thread-3) [    ] 
o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 403375 INFO  (jetty-launcher-2299-thread-3) [    ] 
o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 403375 INFO  (jetty-launcher-2299-thread-3) [    ] 
o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: 
c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 403375 WARN  (jetty-launcher-2299-thread-4) [    ] 
o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 403376 INFO  (jetty-launcher-2299-thread-4) [    ] 
o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 403376 INFO  (jetty-launcher-2299-thread-4) [    ] 
o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 403376 INFO  (jetty-launcher-2299-thread-4) [    ] 
o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: 
c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 403378 INFO  (jetty-launcher-2299-thread-3) [    ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 403378 INFO  (jetty-launcher-2299-thread-3) [    ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 403378 INFO  (jetty-launcher-2299-thread-3) [    ] 
o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 403378 INFO  (jetty-launcher-2299-thread-3) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@407f66e6{/solr,null,AVAILABLE}
   [junit4]   2> 403380 INFO  (jetty-launcher-2299-thread-4) [    ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 403380 INFO  (jetty-launcher-2299-thread-4) [    ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 403380 INFO  (jetty-launcher-2299-thread-4) [    ] 
o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 403380 INFO  (jetty-launcher-2299-thread-4) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@5c16ba9d{/solr,null,AVAILABLE}
   [junit4]   2> 403381 INFO  (jetty-launcher-2299-thread-2) [    ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 403381 INFO  (jetty-launcher-2299-thread-2) [    ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 403381 INFO  (jetty-launcher-2299-thread-2) [    ] 
o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 403382 INFO  (jetty-launcher-2299-thread-3) [    ] 
o.e.j.s.AbstractConnector Started ServerConnector@258b3984{HTTP/1.1,[http/1.1, 
h2c]}{127.0.0.1:41125}
   [junit4]   2> 403382 INFO  (jetty-launcher-2299-thread-3) [    ] 
o.e.j.s.Server Started @403444ms
   [junit4]   2> 403382 INFO  (jetty-launcher-2299-thread-3) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=41125}
   [junit4]   2> 403382 ERROR (jetty-launcher-2299-thread-3) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 403382 INFO  (jetty-launcher-2299-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter Using logger factory 
org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 403383 INFO  (jetty-launcher-2299-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
9.0.0
   [junit4]   2> 403383 INFO  (jetty-launcher-2299-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 403383 INFO  (jetty-launcher-2299-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 403383 INFO  (jetty-launcher-2299-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2019-03-08T16:59:08.496Z
   [junit4]   2> 403385 INFO  (jetty-launcher-2299-thread-4) [    ] 
o.e.j.s.AbstractConnector Started ServerConnector@20bd6443{HTTP/1.1,[http/1.1, 
h2c]}{127.0.0.1:42903}
   [junit4]   2> 403385 INFO  (jetty-launcher-2299-thread-2) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@24cf8fb1{/solr,null,AVAILABLE}
   [junit4]   2> 403385 INFO  (jetty-launcher-2299-thread-4) [    ] 
o.e.j.s.Server Started @403447ms
   [junit4]   2> 403385 INFO  (jetty-launcher-2299-thread-4) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=42903}
   [junit4]   2> 403385 ERROR (jetty-launcher-2299-thread-4) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 403385 INFO  (jetty-launcher-2299-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter Using logger factory 
org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 403385 INFO  (jetty-launcher-2299-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
9.0.0
   [junit4]   2> 403385 INFO  (jetty-launcher-2299-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 403385 INFO  (jetty-launcher-2299-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 403385 INFO  (jetty-launcher-2299-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2019-03-08T16:59:08.498Z
   [junit4]   2> 403389 INFO  (jetty-launcher-2299-thread-2) [    ] 
o.e.j.s.AbstractConnector Started ServerConnector@4b190d39{HTTP/1.1,[http/1.1, 
h2c]}{127.0.0.1:33761}
   [junit4]   2> 403389 INFO  (jetty-launcher-2299-thread-2) [    ] 
o.e.j.s.Server Started @403451ms
   [junit4]   2> 403389 INFO  (jetty-launcher-2299-thread-2) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=33761}
   [junit4]   2> 403390 ERROR (jetty-launcher-2299-thread-2) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 403390 INFO  (jetty-launcher-2299-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter Using logger factory 
org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 403390 INFO  (jetty-launcher-2299-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
9.0.0
   [junit4]   2> 403390 INFO  (jetty-launcher-2299-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 403390 INFO  (jetty-launcher-2299-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 403390 INFO  (jetty-launcher-2299-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2019-03-08T16:59:08.503Z
   [junit4]   2> 403397 WARN  (jetty-launcher-2299-thread-1) [    ] 
o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 403401 INFO  (jetty-launcher-2299-thread-1) [    ] 
o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 403401 INFO  (jetty-launcher-2299-thread-1) [    ] 
o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 403402 INFO  (jetty-launcher-2299-thread-1) [    ] 
o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: 
c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 403410 INFO  (zkConnectionManagerCallback-2301-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 403411 INFO  (jetty-launcher-2299-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 403417 INFO  (jetty-launcher-2299-thread-3) [    ] 
o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay 
is ignored
   [junit4]   2> 403417 INFO  (jetty-launcher-2299-thread-3) [    ] 
o.a.s.c.SolrXmlConfig Configuration parameter 
autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 403418 INFO  (jetty-launcher-2299-thread-3) [    ] 
o.a.s.c.SolrXmlConfig MBean server found: 
com.sun.jmx.mbeanserver.JmxMBeanServer@3664f61, but no JMX reporters were 
configured - adding default JMX reporter.
   [junit4]   2> 403421 INFO  (zkConnectionManagerCallback-2303-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 403421 INFO  (zkConnectionManagerCallback-2305-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 403425 INFO  (jetty-launcher-2299-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 403425 INFO  (jetty-launcher-2299-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 403432 INFO  (jetty-launcher-2299-thread-4) [    ] 
o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay 
is ignored
   [junit4]   2> 403432 INFO  (jetty-launcher-2299-thread-4) [    ] 
o.a.s.c.SolrXmlConfig Configuration parameter 
autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 403433 INFO  (jetty-launcher-2299-thread-2) [    ] 
o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay 
is ignored
   [junit4]   2> 403433 INFO  (jetty-launcher-2299-thread-2) [    ] 
o.a.s.c.SolrXmlConfig Configuration parameter 
autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 403449 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 
0x10312ba5f1d0003, likely client has closed socket
   [junit4]   2> 403502 INFO  (jetty-launcher-2299-thread-2) [    ] 
o.a.s.c.SolrXmlConfig MBean server found: 
com.sun.jmx.mbeanserver.JmxMBeanServer@3664f61, but no JMX reporters were 
configured - adding default JMX reporter.
   [junit4]   2> 403503 INFO  (jetty-launcher-2299-thread-1) [    ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 403503 INFO  (jetty-launcher-2299-thread-1) [    ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 403503 INFO  (jetty-launcher-2299-thread-1) [    ] 
o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 403506 INFO  (jetty-launcher-2299-thread-1) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@14f56814{/solr,null,AVAILABLE}
   [junit4]   2> 403506 INFO  (jetty-launcher-2299-thread-4) [    ] 
o.a.s.c.SolrXmlConfig MBean server found: 
com.sun.jmx.mbeanserver.JmxMBeanServer@3664f61, but no JMX reporters were 
configured - adding default JMX reporter.
   [junit4]   2> 403514 INFO  (jetty-launcher-2299-thread-1) [    ] 
o.e.j.s.AbstractConnector Started ServerConnector@48540908{HTTP/1.1,[http/1.1, 
h2c]}{127.0.0.1:43500}
   [junit4]   2> 403514 INFO  (jetty-launcher-2299-thread-1) [    ] 
o.e.j.s.Server Started @403576ms
   [junit4]   2> 403514 INFO  (jetty-launcher-2299-thread-1) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=43500}
   [junit4]   2> 403514 ERROR (jetty-launcher-2299-thread-1) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 403514 INFO  (jetty-launcher-2299-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter Using logger factory 
org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 403515 INFO  (jetty-launcher-2299-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
9.0.0
   [junit4]   2> 403515 INFO  (jetty-launcher-2299-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 403515 INFO  (jetty-launcher-2299-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 403515 INFO  (jetty-launcher-2299-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2019-03-08T16:59:08.628Z
   [junit4]   2> 403561 INFO  (zkConnectionManagerCallback-2307-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 403562 INFO  (jetty-launcher-2299-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 403566 INFO  (jetty-launcher-2299-thread-1) [    ] 
o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay 
is ignored
   [junit4]   2> 403566 INFO  (jetty-launcher-2299-thread-1) [    ] 
o.a.s.c.SolrXmlConfig Configuration parameter 
autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 403568 INFO  (jetty-launcher-2299-thread-1) [    ] 
o.a.s.c.SolrXmlConfig MBean server found: 
com.sun.jmx.mbeanserver.JmxMBeanServer@3664f61, but no JMX reporters were 
configured - adding default JMX reporter.
   [junit4]   2> 403744 INFO  (jetty-launcher-2299-thread-2) [    ] 
o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: 
WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 403753 WARN  (jetty-launcher-2299-thread-2) [    ] 
o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for 
SslContextFactory@4658969d[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 403759 INFO  (jetty-launcher-2299-thread-4) [    ] 
o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: 
WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 403782 WARN  (jetty-launcher-2299-thread-4) [    ] 
o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for 
SslContextFactory@523e02ac[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 403798 WARN  (jetty-launcher-2299-thread-2) [    ] 
o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for 
SslContextFactory@7bdd21a9[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 403799 INFO  (jetty-launcher-2299-thread-2) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:39876/solr
   [junit4]   2> 403816 WARN  (jetty-launcher-2299-thread-4) [    ] 
o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for 
SslContextFactory@5cf2f652[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 403818 INFO  (jetty-launcher-2299-thread-4) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:39876/solr
   [junit4]   2> 403824 INFO  (jetty-launcher-2299-thread-3) [    ] 
o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: 
WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 403835 WARN  (jetty-launcher-2299-thread-3) [    ] 
o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for 
SslContextFactory@6a859ba2[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 403859 INFO  (zkConnectionManagerCallback-2319-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 403859 INFO  (zkConnectionManagerCallback-2315-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 403874 INFO  (jetty-launcher-2299-thread-1) [    ] 
o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: 
WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 403875 WARN  (jetty-launcher-2299-thread-3) [    ] 
o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for 
SslContextFactory@6de78155[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 403876 INFO  (jetty-launcher-2299-thread-3) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:39876/solr
   [junit4]   2> 403877 INFO  (zkConnectionManagerCallback-2323-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 403881 INFO  (zkConnectionManagerCallback-2326-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 403881 INFO  (zkConnectionManagerCallback-2330-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 403882 WARN  (jetty-launcher-2299-thread-1) [    ] 
o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for 
SslContextFactory@15c48d97[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 403886 WARN  (jetty-launcher-2299-thread-1) [    ] 
o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for 
SslContextFactory@33dace39[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 403887 INFO  (jetty-launcher-2299-thread-1) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:39876/solr
   [junit4]   2> 403895 INFO  (zkConnectionManagerCallback-2339-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 403900 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 
0x10312ba5f1d000c, likely client has closed socket
   [junit4]   2> 403925 INFO  (zkConnectionManagerCallback-2341-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 403925 INFO  (zkConnectionManagerCallback-2343-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 403938 INFO  (jetty-launcher-2299-thread-2) 
[n:127.0.0.1:33761_solr    ] o.a.s.c.OverseerElectionContext I am going to be 
the leader 127.0.0.1:33761_solr
   [junit4]   2> 403940 INFO  (jetty-launcher-2299-thread-2) 
[n:127.0.0.1:33761_solr    ] o.a.s.c.Overseer Overseer 
(id=72922610637013002-127.0.0.1:33761_solr-n_0000000000) starting
   [junit4]   2> 403942 INFO  (jetty-launcher-2299-thread-3) 
[n:127.0.0.1:41125_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:41125_solr
   [junit4]   2> 403948 INFO  (zkCallback-2340-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 403951 INFO  (zkCallback-2342-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 403955 INFO  (jetty-launcher-2299-thread-1) 
[n:127.0.0.1:43500_solr    ] o.a.s.c.ZkController Publish 
node=127.0.0.1:43500_solr as DOWN
   [junit4]   2> 403957 INFO  (jetty-launcher-2299-thread-1) 
[n:127.0.0.1:43500_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating 
transient cache for 4 transient cores
   [junit4]   2> 403957 INFO  (jetty-launcher-2299-thread-1) 
[n:127.0.0.1:43500_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:43500_solr
   [junit4]   2> 403961 INFO  (zkCallback-2325-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 403962 INFO  (zkCallback-2340-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 403963 INFO  (zkCallback-2342-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 403971 INFO  (zkCallback-2325-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 404002 INFO  (zkConnectionManagerCallback-2352-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 404004 INFO  (jetty-launcher-2299-thread-2) 
[n:127.0.0.1:33761_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (2)
   [junit4]   2> 404006 INFO  (jetty-launcher-2299-thread-2) 
[n:127.0.0.1:33761_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster 
at 127.0.0.1:39876/solr ready
   [junit4]   2> 404007 INFO  
(OverseerStateUpdate-72922610637013002-127.0.0.1:33761_solr-n_0000000000) 
[n:127.0.0.1:33761_solr    ] o.a.s.c.Overseer Starting to work on the main 
queue : 127.0.0.1:33761_solr
   [junit4]   2> 404008 INFO  (jetty-launcher-2299-thread-2) 
[n:127.0.0.1:33761_solr    ] o.a.s.c.ZkController Publish 
node=127.0.0.1:33761_solr as DOWN
   [junit4]   2> 404011 INFO  (jetty-launcher-2299-thread-2) 
[n:127.0.0.1:33761_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating 
transient cache for 4 transient cores
   [junit4]   2> 404011 INFO  (jetty-launcher-2299-thread-2) 
[n:127.0.0.1:33761_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:33761_solr
   [junit4]   2> 404013 INFO  (zkCallback-2340-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 404017 INFO  (zkCallback-2342-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 404018 INFO  (zkCallback-2325-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 404034 INFO  (jetty-launcher-2299-thread-4) 
[n:127.0.0.1:42903_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (3)
   [junit4]   2> 404038 INFO  (jetty-launcher-2299-thread-4) 
[n:127.0.0.1:42903_solr    ] o.a.s.c.ZkController Publish 
node=127.0.0.1:42903_solr as DOWN
   [junit4]   2> 404039 INFO  (jetty-launcher-2299-thread-4) 
[n:127.0.0.1:42903_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating 
transient cache for 4 transient cores
   [junit4]   2> 404039 INFO  (jetty-launcher-2299-thread-4) 
[n:127.0.0.1:42903_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:42903_solr
   [junit4]   2> 404040 INFO  (zkCallback-2340-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 404042 INFO  (zkCallback-2342-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 404049 INFO  (zkCallback-2322-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 404050 INFO  (zkCallback-2325-thread-2) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 404050 INFO  (jetty-launcher-2299-thread-2) 
[n:127.0.0.1:33761_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system 
collection, keeping metrics history in memory.
   [junit4]   2> 404076 INFO  (zkConnectionManagerCallback-2357-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 404087 INFO  (jetty-launcher-2299-thread-1) 
[n:127.0.0.1:43500_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (4)
   [junit4]   2> 404093 INFO  (zkCallback-2351-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (4)
   [junit4]   2> 404093 INFO  (zkConnectionManagerCallback-2362-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 404096 INFO  (jetty-launcher-2299-thread-1) 
[n:127.0.0.1:43500_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster 
at 127.0.0.1:39876/solr ready
   [junit4]   2> 404097 INFO  (jetty-launcher-2299-thread-3) 
[n:127.0.0.1:41125_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (4)
   [junit4]   2> 404099 INFO  (jetty-launcher-2299-thread-3) 
[n:127.0.0.1:41125_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster 
at 127.0.0.1:39876/solr ready
   [junit4]   2> 404103 INFO  (jetty-launcher-2299-thread-1) 
[n:127.0.0.1:43500_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system 
collection, keeping metrics history in memory.
   [junit4]   2> 404110 INFO  (jetty-launcher-2299-thread-3) 
[n:127.0.0.1:41125_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system 
collection, keeping metrics history in memory.
   [junit4]   2> 404122 INFO  (zkConnectionManagerCallback-2368-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 404124 INFO  (jetty-launcher-2299-thread-4) 
[n:127.0.0.1:42903_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (4)
   [junit4]   2> 404125 INFO  (jetty-launcher-2299-thread-2) 
[n:127.0.0.1:33761_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@3664f61
   [junit4]   2> 404126 INFO  (jetty-launcher-2299-thread-4) 
[n:127.0.0.1:42903_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster 
at 127.0.0.1:39876/solr ready
   [junit4]   2> 404127 INFO  (jetty-launcher-2299-thread-4) 
[n:127.0.0.1:42903_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system 
collection, keeping metrics history in memory.
   [junit4]   2> 404211 INFO  (jetty-launcher-2299-thread-2) 
[n:127.0.0.1:33761_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@3664f61
   [junit4]   2> 404211 INFO  (jetty-launcher-2299-thread-2) 
[n:127.0.0.1:33761_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@3664f61
   [junit4]   2> 404218 INFO  (jetty-launcher-2299-thread-2) 
[n:127.0.0.1:33761_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.api.collections.HdfsCollectionsAPIDistributedZkTest_A23D00C6F401CC94-001/tempDir-002/node2/.
   [junit4]   2> 404237 INFO  (jetty-launcher-2299-thread-1) 
[n:127.0.0.1:43500_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@3664f61
   [junit4]   2> 404255 INFO  (jetty-launcher-2299-thread-3) 
[n:127.0.0.1:41125_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@3664f61
   [junit4]   2> 404306 INFO  (jetty-launcher-2299-thread-4) 
[n:127.0.0.1:42903_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@3664f61
   [junit4]   2> 404371 INFO  (jetty-launcher-2299-thread-3) 
[n:127.0.0.1:41125_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@3664f61
   [junit4]   2> 404371 INFO  (jetty-launcher-2299-thread-3) 
[n:127.0.0.1:41125_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@3664f61
   [junit4]   2> 404373 INFO  (jetty-launcher-2299-thread-1) 
[n:127.0.0.1:43500_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@3664f61
   [junit4]   2> 404373 INFO  (jetty-launcher-2299-thread-1) 
[n:127.0.0.1:43500_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@3664f61
   [junit4]   2> 404384 INFO  (jetty-launcher-2299-thread-3) 
[n:127.0.0.1:41125_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.api.collections.HdfsCollectionsAPIDistributedZkTest_A23D00C6F401CC94-001/tempDir-002/node3/.
   [junit4]   2> 404386 INFO  (jetty-launcher-2299-thread-1) 
[n:127.0.0.1:43500_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.api.collections.HdfsCollectionsAPIDistributedZkTest_A23D00C6F401CC94-001/tempDir-002/node1/.
   [junit4]   2> 404405 INFO  (jetty-launcher-2299-thread-4) 
[n:127.0.0.1:42903_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@3664f61
   [junit4]   2> 404405 INFO  (jetty-launcher-2299-thread-4) 
[n:127.0.0.1:42903_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@3664f61
   [junit4]   2> 404411 INFO  (jetty-launcher-2299-thread-4) 
[n:127.0.0.1:42903_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.api.collections.HdfsCollectionsAPIDistributedZkTest_A23D00C6F401CC94-001/tempDir-002/node4/.
   [junit4]   2> 404675 INFO  (zkConnectionManagerCallback-2376-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 404677 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testCreationAndDeletion-seed#[A23D00C6F401CC94])
 [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 404678 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testCreationAndDeletion-seed#[A23D00C6F401CC94])
 [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 
127.0.0.1:39876/solr ready
   [junit4]   2> 404698 INFO  (qtp1512331531-6393) [n:127.0.0.1:41125_solr    ] 
o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params 
collection.configName=conf&name=created_and_deleted&nrtReplicas=1&action=CREATE&numShards=1&wt=javabin&version=2
 and sendToOCPQueue=true
   [junit4]   2> 404713 INFO  
(OverseerThreadFactory-2385-thread-1-processing-n:127.0.0.1:33761_solr) 
[n:127.0.0.1:33761_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection 
created_and_deleted
   [junit4]   2> 404844 INFO  
(OverseerStateUpdate-72922610637013002-127.0.0.1:33761_solr-n_0000000000) 
[n:127.0.0.1:33761_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"created_and_deleted",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"created_and_deleted_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:41125/solr";,
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 405050 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr    
x:created_and_deleted_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core 
create command 
qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf&newCollection=true&name=created_and_deleted_shard1_replica_n1&action=CREATE&numShards=1&collection=created_and_deleted&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 405050 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr    
x:created_and_deleted_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault 
Allocating transient cache for 4 transient cores
   [junit4]   2> 406086 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene 
MatchVersion: 9.0.0
   [junit4]   2> 406108 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.s.IndexSchema 
[created_and_deleted_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 406151 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema 
minimal/1.1 with uniqueid field id
   [junit4]   2> 406151 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.c.CoreContainer Creating 
SolrCore 'created_and_deleted_shard1_replica_n1' using configuration from 
collection created_and_deleted, trusted=true
   [junit4]   2> 406151 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX 
monitoring for 'solr.core.created_and_deleted.shard1.replica_n1' (registry 
'solr.core.created_and_deleted.shard1.replica_n1') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@3664f61
   [junit4]   2> 406247 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory 
solr.hdfs.home=hdfs://localhost:36552/solr_hdfs_home
   [junit4]   2> 406247 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Solr 
Kerberos Authentication disabled
   [junit4]   2> 406247 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.c.SolrCore 
[[created_and_deleted_shard1_replica_n1] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.api.collections.HdfsCollectionsAPIDistributedZkTest_A23D00C6F401CC94-001/tempDir-002/node3/created_and_deleted_shard1_replica_n1],
 
dataDir=[hdfs://localhost:36552/solr_hdfs_home/created_and_deleted/core_node2/data/]
   [junit4]   2> 406273 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating 
directory factory for path 
hdfs://localhost:36552/solr_hdfs_home/created_and_deleted/core_node2/data/snapshot_metadata
   [junit4]   2> 406347 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of 
slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 406347 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Block 
cache target memory usage, slab size of [4194304] will allocate [1] slabs and 
use ~[4194304] bytes
   [junit4]   2> 406347 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Creating 
new global HDFS BlockCache
   [junit4]   2> 407299 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.s.b.BlockDirectory Block cache 
on write is disabled
   [junit4]   2> 407310 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating 
directory factory for path 
hdfs://localhost:36552/solr_hdfs_home/created_and_deleted/core_node2/data
   [junit4]   2> 407345 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating 
directory factory for path 
hdfs://localhost:36552/solr_hdfs_home/created_and_deleted/core_node2/data/index
   [junit4]   2> 407352 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of 
slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 407352 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Block 
cache target memory usage, slab size of [4194304] will allocate [1] slabs and 
use ~[4194304] bytes
   [junit4]   2> 407367 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.s.b.BlockDirectory Block cache 
on write is disabled
   [junit4]   2> 408810 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog 
implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 408810 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.u.UpdateLog Initializing 
UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 
maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 408810 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.u.HdfsUpdateLog Initializing 
HdfsUpdateLog: tlogDfsReplication=3
   [junit4]   2> 408859 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: 
disabled
   [junit4]   2> 408859 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: 
disabled
   [junit4]   2> 409071 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening 
[Searcher@7a34cc0c[created_and_deleted_shard1_replica_n1] main]
   [junit4]   2> 409073 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.r.ManagedResourceStorage 
Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 409074 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded 
null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 409078 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits 
will be reserved for 10000ms.
   [junit4]   2> 409081 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max 
version in index or recent updates, using new clock 1627457432263327744
   [junit4]   2> 409088 INFO  
(searcherExecutor-2400-thread-1-processing-n:127.0.0.1:41125_solr 
x:created_and_deleted_shard1_replica_n1 c:created_and_deleted s:shard1 
r:core_node2) [n:127.0.0.1:41125_solr c:created_and_deleted s:shard1 
r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.c.SolrCore 
[created_and_deleted_shard1_replica_n1] Registered new searcher 
Searcher@7a34cc0c[created_and_deleted_shard1_replica_n1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 409094 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update 
of terms at /collections/created_and_deleted/terms/shard1 to 
Terms{values={core_node2=0}, version=0}
   [junit4]   2> 409094 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase 
make sure parent is created /collections/created_and_deleted/leaders/shard1
   [junit4]   2> 409103 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext 
Enough replicas found to continue.
   [junit4]   2> 409103 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I 
may be the new leader - try and sync
   [junit4]   2> 409103 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to 
http://127.0.0.1:41125/solr/created_and_deleted_shard1_replica_n1/
   [junit4]   2> 409104 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - 
now sync replicas to me
   [junit4]   2> 409104 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.c.SyncStrategy 
http://127.0.0.1:41125/solr/created_and_deleted_shard1_replica_n1/ has no 
replicas
   [junit4]   2> 409104 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase 
Creating leader registration node 
/collections/created_and_deleted/leaders/shard1/leader after winning as 
/collections/created_and_deleted/leader_elect/shard1/election/72922610637013005-core_node2-n_0000000000
   [junit4]   2> 409108 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I 
am the new leader: 
http://127.0.0.1:41125/solr/created_and_deleted_shard1_replica_n1/ shard1
   [junit4]   2> 409111 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.c.ZkController I am the leader, 
no recovery necessary
   [junit4]   2> 409113 INFO  (zkCallback-2340-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/created_and_deleted/state.json] for collection 
[created_and_deleted] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 409120 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr 
c:created_and_deleted s:shard1 r:core_node2 
x:created_and_deleted_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] 
webapp=null path=/admin/cores 
params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf&newCollection=true&name=created_and_deleted_shard1_replica_n1&action=CREATE&numShards=1&collection=created_and_deleted&shard=shard1&wt=javabin&version=2&replicaType=NRT}
 status=0 QTime=4072
   [junit4]   2> 409128 INFO  (qtp1512331531-6393) [n:127.0.0.1:41125_solr    ] 
o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 
45 seconds. Check all shard replicas
   [junit4]   2> 409217 INFO  (zkCallback-2340-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/created_and_deleted/state.json] for collection 
[created_and_deleted] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 409218 INFO  (qtp1512331531-6393) [n:127.0.0.1:41125_solr    ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={collection.configName=conf&name=created_and_deleted&nrtReplicas=1&action=CREATE&numShards=1&wt=javabin&version=2}
 status=0 QTime=4520
   [junit4]   2> 409219 INFO  (zkCallback-2340-thread-2) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/created_and_deleted/state.json] for collection 
[created_and_deleted] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 409222 INFO  (qtp1512331531-6396) [n:127.0.0.1:41125_solr    ] 
o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params 
action=LIST&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 409222 INFO  (qtp1512331531-6396) [n:127.0.0.1:41125_solr    ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={action=LIST&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 409222 INFO  (zkCallback-2340-thread-3) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/created_and_deleted/state.json] for collection 
[created_and_deleted] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 409223 INFO  (qtp1512331531-6392) [n:127.0.0.1:41125_solr    ] 
o.a.s.h.a.CollectionsHandler Invoked Collection Action :delete with params 
name=created_and_deleted&action=DELETE&wt=javabin&version=2 and 
sendToOCPQueue=true
   [junit4]   2> 409225 INFO  
(OverseerCollectionConfigSetProcessor-72922610637013002-127.0.0.1:33761_solr-n_0000000000)
 [n:127.0.0.1:33761_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: 
/overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may 
have disconnected from ZooKeeper
   [junit4]   2> 409228 INFO  
(OverseerThreadFactory-2385-thread-2-processing-n:127.0.0.1:33761_solr) 
[n:127.0.0.1:33761_solr    ] o.a.s.c.a.c.OverseerCollectionMessageHandler 
Executing Collection 
Cmd=action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true&deleteMetricsHistory=true,
 asyncId=null
   [junit4]   2> 409230 INFO  (qtp1512331531-6394) [n:127.0.0.1:41125_solr    
x:created_and_deleted_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing 
metric reporters for registry=solr.core.created_and_deleted.shard1.replica_n1, 
tag=null
   [junit4]   2> 409230 INFO  (qtp1512331531-6394) [n:127.0.0.1:41125_solr    
x:created_and_deleted_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing 
reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@68eff5cb: rootName 
= null, domain = solr.core.created_and_deleted.shard1.replica_n1, service url = 
null, agent id = null] for registry 
solr.core.created_and_deleted.shard1.replica_n1 / 
com.codahale.metrics.MetricRegistry@5c87766a
   [junit4]   2> 409259 INFO  (qtp1512331531-6394) [n:127.0.0.1:41125_solr    
x:created_and_deleted_shard1_replica_n1] o.a.s.c.SolrCore 
[created_and_deleted_shard1_replica_n1]  CLOSING SolrCore 
org.apache.solr.core.SolrCore@18dba612
   [junit4]   2> 409259 INFO  (qtp1512331531-6394) [n:127.0.0.1:41125_solr    
x:created_and_deleted_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing 
metric reporters for registry=solr.core.created_and_deleted.shard1.replica_n1, 
tag=18dba612
   [junit4]   2> 409259 INFO  (qtp1512331531-6394) [n:127.0.0.1:41125_solr    
x:created_and_deleted_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing 
metric reporters for 
registry=solr.collection.created_and_deleted.shard1.leader, tag=18dba612
   [junit4]   2> 409318 INFO  (qtp1512331531-6394) [n:127.0.0.1:41125_solr    
x:created_and_deleted_shard1_replica_n1] o.a.s.s.h.HdfsDirectory Closing hdfs 
directory 
hdfs://localhost:36552/solr_hdfs_home/created_and_deleted/core_node2/data/index
   [junit4]   2> 409329 INFO  (qtp1512331531-6394) [n:127.0.0.1:41125_solr    
x:created_and_deleted_shard1_replica_n1] o.a.s.s.h.HdfsDirectory Closing hdfs 
directory 
hdfs://localhost:36552/solr_hdfs_home/created_and_deleted/core_node2/data/snapshot_metadata
   [junit4]   2> 409329 INFO  (qtp1512331531-6394) [n:127.0.0.1:41125_solr    
x:created_and_deleted_shard1_replica_n1] o.a.s.s.h.HdfsDirectory Closing hdfs 
directory 
hdfs://localhost:36552/solr_hdfs_home/created_and_deleted/core_node2/data
   [junit4]   2> 409367 INFO  (qtp1512331531-6394) [n:127.0.0.1:41125_solr    
x:created_and_deleted_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update 
of terms at /collections/created_and_deleted/terms/shard1 to Terms{values={}, 
version=1}
   [junit4]   2> 409370 INFO  (qtp1512331531-6394) [n:127.0.0.1:41125_solr    
x:created_and_deleted_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] 
webapp=null path=/admin/cores 
params={deleteInstanceDir=true&deleteMetricsHistory=true&core=created_and_deleted_shard1_replica_n1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2}
 status=0 QTime=140
   [junit4]   2> 409474 INFO  (zkCallback-2340-thread-3) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDeleted 
path:/collections/created_and_deleted/state.json] for collection 
[created_and_deleted] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 409474 INFO  (zkCallback-2340-thread-2) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDeleted 
path:/collections/created_and_deleted/state.json] for collection 
[created_and_deleted] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 409474 INFO  (zkCallback-2340-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDeleted 
path:/collections/created_and_deleted/state.json] for collection 
[created_and_deleted] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 409474 INFO  (zkCallback-2325-thread-2) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDeleted 
path:/collections/created_and_deleted/state.json] for collection 
[created_and_deleted] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 409494 INFO  (qtp1512331531-6392) [n:127.0.0.1:41125_solr    ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={name=created_and_deleted&action=DELETE&wt=javabin&version=2} status=0 
QTime=271
   [junit4]   2> 409496 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr    ] 
o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params 
action=LIST&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 409496 INFO  (qtp1512331531-6395) [n:127.0.0.1:41125_solr    ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={action=LIST&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 409507 INFO  (jetty-closer-2378-thread-1) [    ] 
o.a.s.c.CoreContainer Shutting down CoreContainer instance=750012282
   [junit4]   2> 409507 INFO  (jetty-closer-2378-thread-1) [    ] 
o.a.s.c.ZkController Remove node as live in 
ZooKeeper:/live_nodes/127.0.0.1:33761_solr
   [junit4]   2> 409513 INFO  (jetty-closer-2378-thread-1) [    ] 
o.a.s.c.CoreContainer Publish this node as DOWN...
   [junit4]   2> 409513 INFO  (jetty-closer-2378-thread-1) [    ] 
o.a.s.c.ZkController Publish node=127.0.0.1:33761_solr as DOWN
   [junit4]   2> 409513 INFO  (jetty-closer-2378-thread-3) [    ] 
o.a.s.c.CoreContainer Shutting down CoreContainer instance=1950560799
   [junit4]   2> 409513 INFO  (jetty-closer-2378-thread-3) [    ] 
o.a.s.c.ZkController Remove node as live in 
ZooKeeper:/live_nodes/127.0.0.1:42903_solr
   [junit4]   2> 409516 INFO  (jetty-closer-2378-thread-4) [    ] 
o.a.s.c.CoreContainer Shutting down CoreContainer instance=528523295
   [junit4]   2> 409516 INFO  (jetty-closer-2378-thread-4) [    ] 
o.a.s.c.ZkController Remove node as live in 
ZooKeeper:/live_nodes/127.0.0.1:43500_solr
   [junit4]   2> 409516 INFO  (jetty-closer-2378-thread-2) [    ] 
o.a.s.c.CoreContainer Shutting down CoreContainer instance=1475976209
   [junit4]   2> 409516 INFO  (jetty-closer-2378-thread-2) [    ] 
o.a.s.c.ZkController Remove node as live in 
ZooKeeper:/live_nodes/127.0.0.1:41125_solr
   [junit4]   2> 409517 INFO  (jetty-closer-2378-thread-4) [    ] 
o.a.s.c.CoreContainer Publish this node as DOWN...
   [junit4]   2> 409517 INFO  (jetty-closer-2378-thread-4) [    ] 
o.a.s.c.ZkController Publish node=127.0.0.1:43500_solr as DOWN
   [junit4]   2> 409517 INFO  (jetty-closer-2378-thread-3) [    ] 
o.a.s.c.CoreContainer Publish this node as DOWN...
   [junit4]   2> 409517 INFO  (jetty-closer-2378-thread-2) [    ] 
o.a.s.c.CoreContainer Publish this node as DOWN...
   [junit4]   2> 409517 INFO  (jetty-closer-2378-thread-2) [    ] 
o.a.s.c.ZkController Publish node=127.0.0.1:41125_solr as DOWN
   [junit4]   2> 409517 INFO  (jetty-closer-2378-thread-3) [    ] 
o.a.s.c.ZkController Publish node=127.0.0.1:42903_solr as DOWN
   [junit4]   2> 409530 INFO  (jetty-closer-2378-thread-1) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, 
tag=null
   [junit4]   2> 409530 INFO  (jetty-closer-2378-thread-1) [    ] 
o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@34bae6bb: rootName = null, 
domain = solr.node, service url = null, agent id = null] for registry solr.node 
/ com.codahale.metrics.MetricRegistry@6ee21d8e
   [junit4]   2> 409538 INFO  (jetty-closer-2378-thread-3) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, 
tag=null
   [junit4]   2> 409538 INFO  (jetty-closer-2378-thread-3) [    ] 
o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@2c642080: rootName = null, 
domain = solr.node, service url = null, agent id = null] for registry solr.node 
/ com.codahale.metrics.MetricRegistry@55b70618
   [junit4]   2> 409540 INFO  (jetty-closer-2378-thread-1) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, 
tag=null
   [junit4]   2> 409540 INFO  (jetty-closer-2378-thread-1) [    ] 
o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@543bb4ed: rootName = null, 
domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / 
com.codahale.metrics.MetricRegistry@53998627
   [junit4]   2> 409550 INFO  (jetty-closer-2378-thread-1) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, 
tag=null
   [junit4]   2> 409550 INFO  (jetty-closer-2378-thread-1) [    ] 
o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@5b7cf20e: rootName = null, 
domain = solr.jetty, service url = null, agent id = null] for registry 
solr.jetty / com.codahale.metrics.MetricRegistry@78399ce0
   [junit4]   2> 409550 INFO  (jetty-closer-2378-thread-1) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, 
tag=null
   [junit4]   2> 409552 INFO  (jetty-closer-2378-thread-3) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, 
tag=null
   [junit4]   2> 409552 INFO  (jetty-closer-2378-thread-3) [    ] 
o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@69f73d51: rootName = null, 
domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / 
com.codahale.metrics.MetricRegistry@53998627
   [junit4]   2> 409553 INFO  (jetty-closer-2378-thread-4) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, 
tag=null
   [junit4]   2> 409553 INFO  (jetty-closer-2378-thread-4) [    ] 
o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@1194466e: rootName = null, 
domain = solr.node, service url = null, agent id = null] for registry solr.node 
/ com.codahale.metrics.MetricRegistry@556d5710
   [junit4]   2> 409554 INFO  (jetty-closer-2378-thread-4) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, 
tag=null
   [junit4]   2> 409554 INFO  (jetty-closer-2378-thread-4) [    ] 
o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@59333640: rootName = null, 
domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / 
com.codahale.metrics.MetricRegistry@53998627
   [junit4]   2> 409556 INFO  (jetty-closer-2378-thread-4) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, 
tag=null
   [junit4]   2> 409556 INFO  (jetty-closer-2378-thread-4) [    ] 
o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@75e5ff84: rootName = null, 
domain = solr.jetty, service url = null, agent id = null] for registry 
solr.jetty / com.codahale.metrics.MetricRegistry@78399ce0
   [junit4]   2> 409556 INFO  (jetty-closer-2378-thread-4) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, 
tag=null
   [junit4]   2> 409560 INFO  (jetty-closer-2378-thread-3) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, 
tag=null
   [junit4]   2> 409560 INFO  (jetty-closer-2378-thread-3) [    ] 
o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@2b1be72e: rootName = null, 
domain = solr.jetty, service url = null, agent id = null] for registry 
solr.jetty / com.codahale.metrics.MetricRegistry@78399ce0
   [junit4]   2> 409561 INFO  (jetty-closer-2378-thread-3) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, 
tag=null
   [junit4]   2> 409569 INFO  (closeThreadPool-2388-thread-2) [    ] 
o.a.s.c.Overseer Overseer 
(id=72922610637013002-127.0.0.1:33761_solr-n_0000000000) closing
   [junit4]   2> 409570 INFO  (jetty-closer-2378-thread-2) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, 
tag=null
   [junit4]   2> 409570 INFO  (jetty-closer-2378-thread-2) [    ] 
o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@7040d5ee: rootName = null, 
domain = solr.node, service url = null, agent id = null] for registry solr.node 
/ com.codahale.metrics.MetricRegistry@435eb3e5
   [junit4]   2> 409570 INFO  
(OverseerStateUpdate-72922610637013002-127.0.0.1:33761_solr-n_0000000000) 
[n:127.0.0.1:33761_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 
127.0.0.1:33761_solr
   [junit4]   2> 409587 INFO  
(OverseerAutoScalingTriggerThread-72922610637013002-127.0.0.1:33761_solr-n_0000000000)
 [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we 
are closed, exiting.
   [junit4]   2> 409587 INFO  (jetty-closer-2378-thread-2) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, 
tag=null
   [junit4]   2> 409587 INFO  (jetty-closer-2378-thread-2) [    ] 
o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@7650d1f0: rootName = null, 
domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / 
com.codahale.metrics.MetricRegistry@53998627
   [junit4]   2> 409588 INFO  (jetty-closer-2378-thread-2) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, 
tag=null
   [junit4]   2> 409588 INFO  (jetty-closer-2378-thread-2) [    ] 
o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@570451f1: rootName = null, 
domain = solr.jetty, service url = null, agent id = null] for registry 
solr.jetty / com.codahale.metrics.MetricRegistry@78399ce0
   [junit4]   2> 409588 INFO  (jetty-closer-2378-thread-2) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, 
tag=null
   [junit4]   2> 409595 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 
0x10312ba5f1d000a, likely client has closed socket
   [junit4]   2> 409595 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 
0x10312ba5f1d000e, likely client has closed socket
   [junit4]   2> 409595 INFO  (zkCallback-2340-thread-2) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (2)
   [junit4]   2> 409596 INFO  (zkCallback-2361-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (2)
   [junit4]   2> 409596 INFO  (zkCallback-2340-thread-3) [    ] 
o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:41125_solr
   [junit4]   2> 409599 INFO  (zkCallback-2361-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 409626 INFO  (jetty-closer-2378-thread-1) [    ] 
o.a.s.c.Overseer Overseer 
(id=72922610637013002-127.0.0.1:33761_solr-n_0000000000) closing
   [junit4]   2> 409626 INFO  (closeThreadPool-2388-thread-1) [    ] 
o.a.s.c.Overseer Overseer 
(id=72922610637013002-127.0.0.1:33761_solr-n_0000000000) closing
   [junit4]   2> 409630 INFO  (jetty-closer-2378-thread-4) [    ] 
o.e.j.s.AbstractConnector Stopped ServerConnector@48540908{HTTP/1.1,[http/1.1, 
h2c]}{127.0.0.1:0}
   [junit4]   2> 409641 INFO  (jetty-closer-2378-thread-4) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@14f56814{/solr,null,UNAVAILABLE}
   [junit4]   2> 409665 INFO  (jetty-closer-2378-thread-2) [    ] 
o.e.j.s.AbstractConnector Stopped ServerConnector@258b3984{HTTP/1.1,[http/1.1, 
h2c]}{127.0.0.1:0}
   [junit4]   2> 409676 INFO  (jetty-closer-2378-thread-4) [    ] 
o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 409676 INFO  (jetty-closer-2378-thread-3) [    ] 
o.e.j.s.AbstractConnector Stopped ServerConnector@20bd6443{HTTP/1.1,[http/1.1, 
h2c]}{127.0.0.1:0}
   [junit4]   2> 409680 INFO  (jetty-closer-2378-thread-1) [    ] 
o.e.j.s.AbstractConnector Stopped ServerConnector@4b190d39{HTTP/1.1,[http/1.1, 
h2c]}{127.0.0.1:0}
   [junit4]   2> 409681 INFO  (jetty-closer-2378-thread-2) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@407f66e6{/solr,null,UNAVAILABLE}
   [junit4]   2> 409681 INFO  (jetty-closer-2378-thread-3) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@5c16ba9d{/solr,null,UNAVAILABLE}
   [junit4]   2> 409697 INFO  (jetty-closer-2378-thread-3) [    ] 
o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 409697 INFO  (jetty-closer-2378-thread-1) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@24cf8fb1{/solr,null,UNAVAILABLE}
   [junit4]   2> 409697 INFO  (jetty-closer-2378-thread-2) [    ] 
o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 409700 INFO  (jetty-closer-2378-thread-1) [    ] 
o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 409710 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testCreationAndDeletion-seed#[A23D00C6F401CC94])
 [    ] o.a.s.c.ZkTestServer Shutting down ZkTestServer.
   [junit4]   2> 409712 WARN  (ZkTestServer Run Thread) [    ] 
o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2>        9       /solr/aliases.json
   [junit4]   2>        9       /solr/clusterprops.json
   [junit4]   2>        4       /solr/security.json
   [junit4]   2>        3       
/solr/collections/created_and_deleted/terms/shard1
   [junit4]   2>        2       /solr/collections/created_and_deleted/state.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2>        9       /solr/clusterstate.json
   [junit4]   2>        6       /solr/collections/created_and_deleted/state.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2>        9       /solr/live_nodes
   [junit4]   2>        9       /solr/collections
   [junit4]   2> 
   [junit4]   2> 409720 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testCreationAndDeletion-seed#[A23D00C6F401CC94])
 [    ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:39876
   [junit4]   2> 409720 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testCreationAndDeletion-seed#[A23D00C6F401CC94])
 [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 39876
   [junit4]   2> 409720 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testCreationAndDeletion-seed#[A23D00C6F401CC94])
 [    ] o.a.s.SolrTestCaseJ4 ###Ending testCreationAndDeletion
   [junit4]   2> 409721 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testCreationAndDeletion-seed#[A23D00C6F401CC94])
 [    ] o.a.s.SolrTestCaseJ4 ###Ending testCreationAndDeletion
   [junit4]   2> 409765 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.deletePartiallyCreatedCollection-seed#[A23D00C6F401CC94])
 [    ] o.a.s.SolrTestCaseJ4 ###Starting deletePartiallyCreatedCollection
   [junit4]   2> 409771 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.deletePartiallyCreatedCollection-seed#[A23D00C6F401CC94])
 [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.api.collections.HdfsCollectionsAPIDistributedZkTest_A23D00C6F401CC94-001/tempDir-003
   [junit4]   2> 409771 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.deletePartiallyCreatedCollection-seed#[A23D00C6F401CC94])
 [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 409777 INFO  (ZkTestServer Run Thread) [    ] 
o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 409777 INFO  (ZkTestServer Run Thread) [    ] 
o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 409878 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.deletePartiallyCreatedCollection-seed#[A23D00C6F401CC94])
 [    ] o.a.s.c.ZkTestServer start zk server on port:34709
   [junit4]   2> 409878 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.deletePartiallyCreatedCollection-seed#[A23D00C6F401CC94])
 [    ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:34709
   [junit4]   2> 409878 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.deletePartiallyCreatedCollection-seed#[A23D00C6F401CC94])
 [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 34709
   [junit4]   2> 409945 INFO  (zkConnectionManagerCallback-2392-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 409969 INFO  (zkConnectionManagerCallback-2394-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 409984 INFO  (zkConnectionManagerCallback-2396-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 410009 WARN  (jetty-launcher-2397-thread-2) [    ] 
o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 410010 INFO  (jetty-launcher-2397-thread-2) [    ] 
o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 410010 INFO  (jetty-launcher-2397-thread-2) [    ] 
o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 410010 WARN  (jetty-launcher-2397-thread-1) [    ] 
o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 410010 INFO  (jetty-launcher-2397-thread-2) [    ] 
o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: 
c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 410010 INFO  (jetty-launcher-2397-thread-1) [    ] 
o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 410010 INFO  (jetty-launcher-2397-thread-1) [    ] 
o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 410011 INFO  (jetty-launcher-2397-thread-1) [    ] 
o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: 
c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 410021 WARN  (jetty-launcher-2397-thread-3) [    ] 
o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 410022 INFO  (jetty-launcher-2397-thread-3) [    ] 
o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 410022 INFO  (jetty-launcher-2397-thread-3) [    ] 
o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 410022 INFO  (jetty-launcher-2397-thread-3) [    ] 
o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: 
c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 410073 WARN  (jetty-launcher-2397-thread-4) [    ] 
o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 410091 INFO  (jetty-launcher-2397-thread-4) [    ] 
o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 410091 INFO  (jetty-launcher-2397-thread-4) [    ] 
o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 410091 INFO  (jetty-launcher-2397-thread-4) [    ] 
o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: 
c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 410105 INFO  (jetty-launcher-2397-thread-2) [    ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 410105 INFO  (jetty-launcher-2397-thread-2) [    ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 410105 INFO  (jetty-launcher-2397-thread-2) [    ] 
o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 410106 INFO  (jetty-launcher-2397-thread-2) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@4b58a746{/solr,null,AVAILABLE}
   [junit4]   2> 410194 INFO  (jetty-launcher-2397-thread-3) [    ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 410194 INFO  (jetty-launcher-2397-thread-3) [    ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 410194 INFO  (jetty-launcher-2397-thread-3) [    ] 
o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 410194 INFO  (jetty-launcher-2397-thread-2) [    ] 
o.e.j.s.AbstractConnector Started ServerConnector@7076acd2{HTTP/1.1,[http/1.1, 
h2c]}{127.0.0.1:42515}
   [junit4]   2> 410194 INFO  (jetty-launcher-2397-thread-2) [    ] 
o.e.j.s.Server Started @410256ms
   [junit4]   2> 410194 INFO  (jetty-launcher-2397-thread-2) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=42515}
   [junit4]   2> 410195 ERROR (jetty-launcher-2397-thread-2) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 410195 INFO  (jetty-launcher-2397-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter Using logger factory 
org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 410195 INFO  (jetty-launcher-2397-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
9.0.0
   [junit4]   2> 410195 INFO  (jetty-launcher-2397-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 410195 INFO  (jetty-launcher-2397-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 410195 INFO  (jetty-launcher-2397-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2019-03-08T16:59:15.308Z
   [junit4]   2> 410210 INFO  (jetty-launcher-2397-thread-3) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@7c8eb906{/solr,null,AVAILABLE}
   [junit4]   2> 410224 INFO  (jetty-launcher-2397-thread-4) [    ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 410224 INFO  (jetty-launcher-2397-thread-4) [    ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 410224 INFO  (jetty-launcher-2397-thread-4) [    ] 
o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 410225 INFO  (jetty-launcher-2397-thread-4) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@474eb516{/solr,null,AVAILABLE}
   [junit4]   2> 410226 INFO  (jetty-launcher-2397-thread-1) [    ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 410226 INFO  (jetty-launcher-2397-thread-1) [    ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 410226 INFO  (jetty-launcher-2397-thread-1) [    ] 
o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 410227 INFO  (jetty-launcher-2397-thread-1) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@4732d4da{/solr,null,AVAILABLE}
   [junit4]   2> 410227 INFO  (jetty-launcher-2397-thread-4) [    ] 
o.e.j.s.AbstractConnector Started ServerConnector@28bc8065{HTTP/1.1,[http/1.1, 
h2c]}{127.0.0.1:36639}
   [junit4]   2> 410227 INFO  (jetty-launcher-2397-thread-4) [    ] 
o.e.j.s.Server Started @410289ms
   [junit4]   2> 410227 INFO  (jetty-launcher-2397-thread-4) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=36639}
   [junit4]   2> 410227 ERROR (jetty-launcher-2397-thread-4) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 410228 INFO  (jetty-launcher-2397-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter Using logger factory 
org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 410228 INFO  (jetty-launcher-2397-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
9.0.0
   [junit4]   2> 410228 INFO  (jetty-launcher-2397-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 410228 INFO  (jetty-launcher-2397-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 410228 INFO  (jetty-launcher-2397-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2019-03-08T16:59:15.341Z
   [junit4]   2> 410273 INFO  (zkConnectionManagerCallback-2401-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 410273 INFO  (jetty-launcher-2397-thread-1) [    ] 
o.e.j.s.AbstractConnector Started ServerConnector@45b901bd{HTTP/1.1,[http/1.1, 
h2c]}{127.0.0.1:39722}
   [junit4]   2> 410273 INFO  (jetty-launcher-2397-thread-1) [    ] 
o.e.j.s.Server Started @410335ms
   [junit4]   2> 410273 INFO  (jetty-launcher-2397-thread-1) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=39722}
   [junit4]   2> 410273 ERROR (jetty-launcher-2397-thread-1) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 410273 INFO  (jetty-launcher-2397-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter Using logger factory 
org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 410273 INFO  (jetty-launcher-2397-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
9.0.0
   [junit4]   2> 410273 INFO  (jetty-launcher-2397-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 410273 INFO  (jetty-launcher-2397-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 410273 INFO  (jetty-launcher-2397-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2019-03-08T16:59:15.386Z
   [junit4]   2> 410275 INFO  (jetty-launcher-2397-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 410280 INFO  (jetty-launcher-2397-thread-4) [    ] 
o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay 
is ignored
   [junit4]   2> 410280 INFO  (jetty-launcher-2397-thread-4) [    ] 
o.a.s.c.SolrXmlConfig Configuration parameter 
autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 410281 INFO  (jetty-launcher-2397-thread-3) [    ] 
o.e.j.s.AbstractConnector Started ServerConnector@74fc1d6c{HTTP/1.1,[http/1.1, 
h2c]}{127.0.0.1:41647}
   [junit4]   2> 410281 INFO  (jetty-launcher-2397-thread-3) [    ] 
o.e.j.s.Server Started @410343ms
   [junit4]   2> 410281 INFO  (jetty-launcher-2397-thread-3) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=41647}
   [junit4]   2> 410281 ERROR (jetty-launcher-2397-thread-3) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 410281 INFO  (jetty-launcher-2397-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter Using logger factory 
org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 410281 INFO  (jetty-launcher-2397-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
9.0.0
   [junit4]   2> 410281 INFO  (jetty-launcher-2397-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 410281 INFO  (jetty-launcher-2397-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 410282 INFO  (jetty-launcher-2397-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2019-03-08T16:59:15.394Z
   [junit4]   2> 410283 INFO  (zkConnectionManagerCallback-2399-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 410284 INFO  (jetty-launcher-2397-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 410287 INFO  (jetty-launcher-2397-thread-2) [    ] 
o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay 
is ignored
   [junit4]   2> 410287 INFO  (jetty-launcher-2397-thread-2) [    ] 
o.a.s.c.SolrXmlConfig Configuration parameter 
autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 410290 INFO  (jetty-launcher-2397-thread-2) [    ] 
o.a.s.c.SolrXmlConfig MBean server found: 
com.sun.jmx.mbeanserver.JmxMBeanServer@3664f61, but no JMX reporters were 
configured - adding default JMX reporter.
   [junit4]   2> 410291 INFO  (jetty-launcher-2397-thread-4) [    ] 
o.a.s.c.SolrXmlConfig MBean server found: 
com.sun.jmx.mbeanserver.JmxMBeanServer@3664f61, but no JMX reporters were 
configured - adding default JMX reporter.
   [junit4]   2> 410295 INFO  (zkConnectionManagerCallback-2403-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 410301 INFO  (jetty-launcher-2397-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 410304 INFO  (jetty-launcher-2397-thread-1) [    ] 
o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay 
is ignored
   [junit4]   2> 410306 INFO  (jetty-launcher-2397-thread-1) [    ] 
o.a.s.c.SolrXmlConfig Configuration parameter 
autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 410307 INFO  (jetty-launcher-2397-thread-1) [    ] 
o.a.s.c.SolrXmlConfig MBean server found: 
com.sun.jmx.mbeanserver.JmxMBeanServer@3664f61, but no JMX reporters were 
configured - adding default JMX reporter.
   [junit4]   2> 410311 INFO  (zkConnectionManagerCallback-2405-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 410320 INFO  (jetty-launcher-2397-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 410330 INFO  (jetty-launcher-2397-thread-3) [    ] 
o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay 
is ignored
   [junit4]   2> 410330 INFO  (jetty-launcher-2397-thread-3) [    ] 
o.a.s.c.SolrXmlConfig Configuration parameter 
autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 410332 INFO  (jetty-launcher-2397-thread-3) [    ] 
o.a.s.c.SolrXmlConfig MBean server found: 
com.sun.jmx.mbeanserver.JmxMBeanServer@3664f61, but no JMX reporters were 
configured - adding default JMX reporter.
   [junit4]   2> 410595 INFO  (jetty-launcher-2397-thread-3) [    ] 
o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: 
WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 410610 WARN  (jetty-launcher-2397-thread-3) [    ] 
o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for 
SslContextFactory@103722e5[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 410635 WARN  (jetty-launcher-2397-thread-3) [    ] 
o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for 
SslContextFactory@3e3df1b7[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 410636 INFO  (jetty-launcher-2397-thread-3) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34709/solr
   [junit4]   2> 410656 INFO  (zkConnectionManagerCallback-2411-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 410667 INFO  (jetty-launcher-2397-thread-4) [    ] 
o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: 
WhitelistHostChecker [whitelistHosts=null, whitelis

[...truncated too long message...]

at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
   [junit4]   2>         at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:288)
   [junit4]   2>         at 
org.apache.hadoop.ipc.Server$Handler.run(Server.java:2668)
   [junit4]   2>   46) Thread[id=6265, 
name=org.apache.hadoop.hdfs.server.blockmanagement.PendingReconstructionBlocks$PendingReconstructionMonitor@519f9fb7,
 state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]   2>         at java.lang.Thread.sleep(Native Method)
   [junit4]   2>         at 
org.apache.hadoop.hdfs.server.blockmanagement.PendingReconstructionBlocks$PendingReconstructionMonitor.run(PendingReconstructionBlocks.java:246)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>   47) Thread[id=6320, name=Socket Reader #1 for port 37000, 
state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]   2>         at sun.nio.ch.EPollArrayWrapper.interrupt(Native 
Method)
   [junit4]   2>         at 
sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:317)
   [junit4]   2>         at 
sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:207)
   [junit4]   2>         at 
java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:213)
   [junit4]   2>         at 
java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:219)
   [junit4]   2>         at 
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:92)
   [junit4]   2>         at 
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]   2>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]   2>         at 
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
   [junit4]   2>         at 
org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1097)
   [junit4]   2>         at 
org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076)
   [junit4]   2>   48) Thread[id=6327, name=IPC Server handler 0 on 37000, 
state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]   2>         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   [junit4]   2>         at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
   [junit4]   2>         at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:288)
   [junit4]   2>         at 
org.apache.hadoop.ipc.Server$Handler.run(Server.java:2668)
   [junit4]   2>   49) Thread[id=6266, name=DatanodeAdminMonitor-0, 
state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]   2>         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   [junit4]   2>         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
   [junit4]   2>         at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   [junit4]   2>         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   [junit4]   2>         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   [junit4]   2>         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>   50) Thread[id=6352, 
name=refreshUsed-/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.api.collections.HdfsCollectionsAPIDistributedZkTest_A23D00C6F401CC94-001/tempDir-001/hdfsBaseDir/data/data2/current/BP-978842632-10.41.0.5-1552064342193,
 state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]   2>         at java.lang.Thread.sleep(Native Method)
   [junit4]   2>         at 
org.apache.hadoop.fs.CachingGetSpaceUsed$RefreshThread.run(CachingGetSpaceUsed.java:179)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>   51) Thread[id=6331, name=IPC Server handler 4 on 37000, 
state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]   2>         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   [junit4]   2>         at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
   [junit4]   2>         at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:288)
   [junit4]   2>         at 
org.apache.hadoop.ipc.Server$Handler.run(Server.java:2668)
   [junit4]   2>   52) Thread[id=6302, name=nioEventLoopGroup-2-1, 
state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]   2>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native 
Method)
   [junit4]   2>         at 
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
   [junit4]   2>         at 
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
   [junit4]   2>         at 
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]   2>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]   2>         at 
io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
   [junit4]   2>         at 
io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:754)
   [junit4]   2>         at 
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:410)
   [junit4]   2>         at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
   [junit4]   2>         at 
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene80): {}, 
docValues:{}, maxPointsInLeafNode=125, maxMBSortInHeap=7.126346147659913, 
sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@f454086),
 locale=en-NZ, timezone=Asia/Dili
   [junit4]   2> NOTE: Linux 4.4.0-137-generic amd64/Oracle Corporation 
1.8.0_191 (64-bit)/cpus=4,threads=66,free=130241800,total=438829056
   [junit4]   2> NOTE: All tests run in this JVM: [SplitShardTest, 
TestOnReconnectListenerSupport, SolrCloudReportersTest, 
CdcrVersionReplicationTest, TestFileDictionaryLookup, MetricTriggerTest, 
TestBM25SimilarityFactory, MoveReplicaTest, CustomHighlightComponentTest, 
AutoscalingHistoryHandlerTest, TestTrie, FileBasedSpellCheckerTest, 
BJQParserTest, TestMaxScoreQueryParser, TestReload, TestLuceneIndexBackCompat, 
CoreAdminOperationTest, TestComplexPhraseQParserPlugin, TestCoreAdminApis, 
RAMDirectoryFactoryTest, TestSchemalessBufferedUpdates, 
FieldMutatingUpdateProcessorTest, TestFacetMethods, 
ChaosMonkeyNothingIsSafeWithPullReplicasTest, TestGraphMLResponseWriter, 
TestConfigReload, ParsingFieldUpdateProcessorsTest, TestManagedSchema, 
SOLR749Test, DistribJoinFromCollectionTest, TestCollectionAPI, 
ExternalFileFieldSortTest, ChangedSchemaMergeTest, TestRemoteStreaming, 
TestExclusionRuleCollectionAccess, TestConfigSetProperties, 
LeaderElectionIntegrationTest, DirectoryFactoryTest, 
LegacyCloudClusterPropTest, TestSQLHandlerNonCloud, TestSystemIdResolver, 
HdfsCollectionsAPIDistributedZkTest]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 
'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=HdfsCollectionsAPIDistributedZkTest -Dtests.seed=A23D00C6F401CC94 
-Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt
 -Dtests.locale=en-NZ -Dtests.timezone=Asia/Dili -Dtests.asserts=true 
-Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J2 | HdfsCollectionsAPIDistributedZkTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 
3600000 msec).
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([A23D00C6F401CC94]:0)
   [junit4] Completed [357/846 (1!)] on J2 in 3613.55s, 3 tests, 2 errors <<< 
FAILURES!

[...truncated 1463 lines...]
   [junit4] JVM J2: stdout was not empty, see: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/temp/junit4-J2-20190308_165224_976435898618271851762.sysout
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/heapdumps/java_pid18661.hprof
 ...
   [junit4] Heap dump file created [718649182 bytes in 18.978 secs]
   [junit4] <<< JVM J2: EOF ----

[...truncated 9017 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:651:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:585:
 Some of the tests produced a heap dump, but did not fail. Maybe a suppressed 
OutOfMemoryError? Dumps created:
* java_pid18661.hprof

Total time: 281 minutes 25 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to