Build: https://builds.apache.org/job/Lucene-Solr-BadApples-NightlyTests-7.x/46/

2 tests failed.
FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test

Error Message:
Error from server at http://127.0.0.1:35923//control_collection: Cannot talk to 
ZooKeeper - Updates are disabled.

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at http://127.0.0.1:35923//control_collection: Cannot talk to 
ZooKeeper - Updates are disabled.
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:504)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:479)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.commit(AbstractFullDistribZkTestBase.java:1701)
        at 
org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexingBatchPerRequestWithHttpSolrClient(FullSolrCloudDistribCmdsTest.java:658)
        at 
org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test(FullSolrCloudDistribCmdsTest.java:154)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1075)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1047)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.hdfs.HdfsRestartWhileUpdatingTest.test

Error Message:
There are still nodes recoverying - waited for 320 seconds

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 320 
seconds
        at 
__randomizedtesting.SeedInfo.seed([24FFF8DB5625604C:ACABC701F8D90DB4]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:195)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:1014)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1571)
        at 
org.apache.solr.cloud.RestartWhileUpdatingTest.test(RestartWhileUpdatingTest.java:144)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1075)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1047)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 14723 lines...]
   [junit4] Suite: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
   [junit4]   2> 2223403 INFO  
(SUITE-FullSolrCloudDistribCmdsTest-seed#[24FFF8DB5625604C]-worker) [    ] 
o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: 
test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest_24FFF8DB5625604C-001/init-core-data-001
   [junit4]   2> 2223404 WARN  
(SUITE-FullSolrCloudDistribCmdsTest-seed#[24FFF8DB5625604C]-worker) [    ] 
o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=105 numCloses=105
   [junit4]   2> 2223404 INFO  
(SUITE-FullSolrCloudDistribCmdsTest-seed#[24FFF8DB5625604C]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) 
w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 2223407 INFO  
(SUITE-FullSolrCloudDistribCmdsTest-seed#[24FFF8DB5625604C]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: 
@org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 2223407 INFO  
(SUITE-FullSolrCloudDistribCmdsTest-seed#[24FFF8DB5625604C]-worker) [    ] 
o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 2223411 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2223412 INFO  (ZkTestServer Run Thread) [    ] 
o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2223412 INFO  (ZkTestServer Run Thread) [    ] 
o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2223512 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.c.ZkTestServer start zk server on port:34379
   [junit4]   2> 2223512 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:34379
   [junit4]   2> 2223512 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.c.ZkTestServer connecting to 127.0.0.1 34379
   [junit4]   2> 2223517 INFO  (zkConnectionManagerCallback-10333-thread-1) [   
 ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2223521 INFO  (zkConnectionManagerCallback-10335-thread-1) [   
 ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2223522 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.c.ZkTestServer put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
 to /configs/conf1/solrconfig.xml
   [junit4]   2> 2223524 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.c.ZkTestServer put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml
 to /configs/conf1/schema.xml
   [junit4]   2> 2223526 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.c.ZkTestServer put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
 to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2223528 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.c.ZkTestServer put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
 to /configs/conf1/stopwords.txt
   [junit4]   2> 2223530 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.c.ZkTestServer put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt
 to /configs/conf1/protwords.txt
   [junit4]   2> 2223532 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.c.ZkTestServer put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml
 to /configs/conf1/currency.xml
   [junit4]   2> 2223533 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.c.ZkTestServer put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
 to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2223535 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.c.ZkTestServer put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
 to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2223536 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.c.ZkTestServer put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
 to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2223538 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.c.ZkTestServer put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
 to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2223539 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.c.ZkTestServer put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
 to /configs/conf1/synonyms.txt
   [junit4]   2> 2223541 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly 
asked otherwise
   [junit4]   2> 2223765 WARN  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2223765 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2223765 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2223765 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: 
c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2223767 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2223767 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2223767 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 2223767 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@7625299a{/,null,AVAILABLE}
   [junit4]   2> 2223770 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@615ffbcc{HTTP/1.1,[http/1.1]}{127.0.0.1:35923}
   [junit4]   2> 2223770 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.e.j.s.Server Started @2223872ms
   [junit4]   2> 2223770 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest_24FFF8DB5625604C-001/tempDir-001/control/data,
 replicaType=NRT, hostContext=/, hostPort=35923, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest_24FFF8DB5625604C-001/control-001/cores}
   [junit4]   2> 2223770 ERROR 
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 2223771 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.s.SolrDispatchFilter Using logger factory 
org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2223771 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
7.8.0
   [junit4]   2> 2223771 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2223771 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2223771 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2019-01-30T12:01:27.020Z
   [junit4]   2> 2223773 INFO  (zkConnectionManagerCallback-10337-thread-1) [   
 ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2223775 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 2223775 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest_24FFF8DB5625604C-001/control-001/solr.xml
   [junit4]   2> 2223780 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay 
is ignored
   [junit4]   2> 2223780 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.c.SolrXmlConfig Configuration parameter 
autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2223782 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.c.SolrXmlConfig MBean server found: 
com.sun.jmx.mbeanserver.JmxMBeanServer@53e0c7af, but no JMX reporters were 
configured - adding default JMX reporter.
   [junit4]   2> 2224091 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: 
WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 2224097 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34379/solr
   [junit4]   2> 2224100 INFO  (zkConnectionManagerCallback-10341-thread-1) [   
 ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2224103 INFO  (zkConnectionManagerCallback-10343-thread-1) [   
 ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2224223 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) 
[n:127.0.0.1:35923_    ] o.a.s.c.OverseerElectionContext I am going to be the 
leader 127.0.0.1:35923_
   [junit4]   2> 2224224 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) 
[n:127.0.0.1:35923_    ] o.a.s.c.Overseer Overseer 
(id=72711934908760068-127.0.0.1:35923_-n_0000000000) starting
   [junit4]   2> 2224232 INFO  (zkConnectionManagerCallback-10350-thread-1) [   
 ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2224236 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) 
[n:127.0.0.1:35923_    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 
127.0.0.1:34379/solr ready
   [junit4]   2> 2224237 INFO  
(OverseerStateUpdate-72711934908760068-127.0.0.1:35923_-n_0000000000) 
[n:127.0.0.1:35923_    ] o.a.s.c.Overseer Starting to work on the main queue : 
127.0.0.1:35923_
   [junit4]   2> 2224237 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) 
[n:127.0.0.1:35923_    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:35923_
   [junit4]   2> 2224239 INFO  (zkCallback-10349-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2224240 INFO  
(OverseerStateUpdate-72711934908760068-127.0.0.1:35923_-n_0000000000) 
[n:127.0.0.1:35923_    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 2224270 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) 
[n:127.0.0.1:35923_    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, 
keeping metrics history in memory.
   [junit4]   2> 2224298 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) 
[n:127.0.0.1:35923_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@53e0c7af
   [junit4]   2> 2224315 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) 
[n:127.0.0.1:35923_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@53e0c7af
   [junit4]   2> 2224315 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) 
[n:127.0.0.1:35923_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@53e0c7af
   [junit4]   2> 2224317 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) 
[n:127.0.0.1:35923_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions 
underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest_24FFF8DB5625604C-001/control-001/cores
   [junit4]   2> 2224347 INFO  (zkConnectionManagerCallback-10356-thread-1) [   
 ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2224348 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2224349 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34379/solr ready
   [junit4]   2> 2224350 INFO  (qtp1938485682-43112) [n:127.0.0.1:35923_    ] 
o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params 
collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:35923_&wt=javabin&version=2
 and sendToOCPQueue=true
   [junit4]   2> 2224353 INFO  
(OverseerThreadFactory-10492-thread-1-processing-n:127.0.0.1:35923_) 
[n:127.0.0.1:35923_    ] o.a.s.c.a.c.CreateCollectionCmd Create collection 
control_collection
   [junit4]   2> 2224461 INFO  (qtp1938485682-43114) [n:127.0.0.1:35923_    ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics 
params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core}
 status=0 QTime=0
   [junit4]   2> 2224462 INFO  (qtp1938485682-43115) [n:127.0.0.1:35923_    ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics 
params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core}
 status=0 QTime=0
   [junit4]   2> 2224463 INFO  (qtp1938485682-43111) [n:127.0.0.1:35923_    ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics 
params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core}
 status=0 QTime=0
   [junit4]   2> 2224463 INFO  (qtp1938485682-43113) [n:127.0.0.1:35923_    ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics 
params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core}
 status=0 QTime=0
   [junit4]   2> 2224464 INFO  (qtp1938485682-43114) [n:127.0.0.1:35923_    ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics 
params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core}
 status=0 QTime=0
   [junit4]   2> 2224470 INFO  (qtp1938485682-43115) [n:127.0.0.1:35923_    
x:control_collection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core 
create command 
qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2224470 INFO  (qtp1938485682-43115) [n:127.0.0.1:35923_    
x:control_collection_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault 
Allocating transient cache for 4 transient cores
   [junit4]   2> 2225490 INFO  (qtp1938485682-43115) [n:127.0.0.1:35923_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.8.0
   [junit4]   2> 2225524 INFO  (qtp1938485682-43115) [n:127.0.0.1:35923_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 2225605 INFO  (qtp1938485682-43115) [n:127.0.0.1:35923_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 2225627 INFO  (qtp1938485682-43115) [n:127.0.0.1:35923_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' 
using configuration from collection control_collection, trusted=true
   [junit4]   2> 2225627 INFO  (qtp1938485682-43115) [n:127.0.0.1:35923_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.core.control_collection.shard1.replica_n1' (registry 
'solr.core.control_collection.shard1.replica_n1') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@53e0c7af
   [junit4]   2> 2225628 INFO  (qtp1938485682-43115) [n:127.0.0.1:35923_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore 
at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest_24FFF8DB5625604C-001/control-001/cores/control_collection_shard1_replica_n1],
 
dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest_24FFF8DB5625604C-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 2225631 INFO  (qtp1938485682-43115) [n:127.0.0.1:35923_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: 
maxMergeAtOnce=48, maxMergeAtOnceExplicit=31, maxMergedSegmentMB=0.6875, 
floorSegmentMB=2.1064453125, forceMergeDeletesPctAllowed=2.4877563675221106, 
segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=0.2836553724847344, deletesPctAllowed=38.079315558810634
   [junit4]   2> 2225633 WARN  (qtp1938485682-43115) [n:127.0.0.1:35923_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = 
requestHandler,name = /dump,class = DumpRequestHandler,attributes = 
{initParams=a, name=/dump, class=DumpRequestHandler},args = 
{defaults={a=A,b=B}}}
   [junit4]   2> 2225719 INFO  (qtp1938485682-43115) [n:127.0.0.1:35923_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.u.UpdateHandler Using UpdateLog implementation: 
org.apache.solr.update.UpdateLog
   [junit4]   2> 2225720 INFO  (qtp1938485682-43115) [n:127.0.0.1:35923_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2225723 INFO  (qtp1938485682-43115) [n:127.0.0.1:35923_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2225723 INFO  (qtp1938485682-43115) [n:127.0.0.1:35923_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2225724 INFO  (qtp1938485682-43115) [n:127.0.0.1:35923_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.MockRandomMergePolicy: 
org.apache.lucene.index.MockRandomMergePolicy@2ea8defa
   [junit4]   2> 2225725 INFO  (qtp1938485682-43115) [n:127.0.0.1:35923_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@1e8ef4fe[control_collection_shard1_replica_n1] main]
   [junit4]   2> 2225727 INFO  (qtp1938485682-43115) [n:127.0.0.1:35923_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf1
   [junit4]   2> 2225728 INFO  (qtp1938485682-43115) [n:127.0.0.1:35923_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2225729 INFO  (qtp1938485682-43115) [n:127.0.0.1:35923_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2225729 INFO  (qtp1938485682-43115) [n:127.0.0.1:35923_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.u.UpdateLog Could not find max version in index or recent updates, using 
new clock 1624086611469795328
   [junit4]   2> 2225733 INFO  
(searcherExecutor-10497-thread-1-processing-n:127.0.0.1:35923_ 
x:control_collection_shard1_replica_n1 c:control_collection s:shard1) 
[n:127.0.0.1:35923_ c:control_collection s:shard1  
x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore 
[control_collection_shard1_replica_n1] Registered new searcher 
Searcher@1e8ef4fe[control_collection_shard1_replica_n1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2225737 INFO  (qtp1938485682-43115) [n:127.0.0.1:35923_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.ZkShardTerms Successful update of terms at 
/collections/control_collection/terms/shard1 to Terms{values={core_node2=0}, 
version=0}
   [junit4]   2> 2225738 INFO  (qtp1938485682-43115) [n:127.0.0.1:35923_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.ShardLeaderElectionContextBase make sure parent is created 
/collections/control_collection/leaders/shard1
   [junit4]   2> 2225741 INFO  (qtp1938485682-43115) [n:127.0.0.1:35923_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2225741 INFO  (qtp1938485682-43115) [n:127.0.0.1:35923_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2225741 INFO  (qtp1938485682-43115) [n:127.0.0.1:35923_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.SyncStrategy Sync replicas to 
http://127.0.0.1:35923/control_collection_shard1_replica_n1/
   [junit4]   2> 2225741 INFO  (qtp1938485682-43115) [n:127.0.0.1:35923_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2225742 INFO  (qtp1938485682-43115) [n:127.0.0.1:35923_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.SyncStrategy 
http://127.0.0.1:35923/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 2225742 INFO  (qtp1938485682-43115) [n:127.0.0.1:35923_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.ShardLeaderElectionContext Found all replicas participating in 
election, clear LIR
   [junit4]   2> 2225742 INFO  (qtp1938485682-43115) [n:127.0.0.1:35923_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node 
/collections/control_collection/leaders/shard1/leader after winning as 
/collections/control_collection/leader_elect/shard1/election/72711934908760068-core_node2-n_0000000000
   [junit4]   2> 2225744 INFO  (qtp1938485682-43115) [n:127.0.0.1:35923_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.ShardLeaderElectionContext I am the new leader: 
http://127.0.0.1:35923/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 2225846 INFO  (zkCallback-10342-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/control_collection/state.json] for collection 
[control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2225847 INFO  (zkCallback-10342-thread-2) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/control_collection/state.json] for collection 
[control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2225847 INFO  (qtp1938485682-43115) [n:127.0.0.1:35923_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2225850 INFO  (qtp1938485682-43115) [n:127.0.0.1:35923_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT}
 status=0 QTime=1380
   [junit4]   2> 2225858 INFO  (qtp1938485682-43112) [n:127.0.0.1:35923_    ] 
o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 
45 seconds. Check all shard replicas
   [junit4]   2> 2225950 INFO  (zkCallback-10342-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/control_collection/state.json] for collection 
[control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2225950 INFO  (zkCallback-10342-thread-3) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/control_collection/state.json] for collection 
[control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2225950 INFO  (zkCallback-10342-thread-2) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/control_collection/state.json] for collection 
[control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2225951 INFO  (qtp1938485682-43112) [n:127.0.0.1:35923_    ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:35923_&wt=javabin&version=2}
 status=0 QTime=1600
   [junit4]   2> 2225957 INFO  (zkConnectionManagerCallback-10361-thread-1) [   
 ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2225958 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2225959 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34379/solr ready
   [junit4]   2> 2225959 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection 
loss:false
   [junit4]   2> 2225960 INFO  (qtp1938485682-43115) [n:127.0.0.1:35923_    ] 
o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params 
collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=3&createNodeSet=&stateFormat=2&wt=javabin&version=2
 and sendToOCPQueue=true
   [junit4]   2> 2225963 INFO  
(OverseerThreadFactory-10492-thread-2-processing-n:127.0.0.1:35923_) 
[n:127.0.0.1:35923_    ] o.a.s.c.a.c.CreateCollectionCmd Create collection 
collection1
   [junit4]   2> 2225964 INFO  
(OverseerCollectionConfigSetProcessor-72711934908760068-127.0.0.1:35923_-n_0000000000)
 [n:127.0.0.1:35923_    ] o.a.s.c.OverseerTaskQueue Response ZK path: 
/overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may 
have disconnected from ZooKeeper
   [junit4]   2> 2226168 WARN  
(OverseerThreadFactory-10492-thread-2-processing-n:127.0.0.1:35923_) 
[n:127.0.0.1:35923_    ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to 
create a collection (collection1) without cores.
   [junit4]   2> 2226170 INFO  (qtp1938485682-43115) [n:127.0.0.1:35923_    ] 
o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 
45 seconds. Check all shard replicas
   [junit4]   2> 2226171 INFO  (qtp1938485682-43115) [n:127.0.0.1:35923_    ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=3&createNodeSet=&stateFormat=2&wt=javabin&version=2}
 status=0 QTime=210
   [junit4]   2> 2226172 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.c.AbstractFullDistribZkTestBase Creating jetty instances 
pullReplicaCount=0 numOtherReplicas=6
   [junit4]   2> 2226350 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest_24FFF8DB5625604C-001/shard-1-001
 of type NRT
   [junit4]   2> 2226351 WARN  (closeThreadPool-10362-thread-1) [    ] 
o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2226351 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2226351 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2226351 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: 
c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2226353 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2226353 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2226353 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 2226353 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@55da7751{/,null,AVAILABLE}
   [junit4]   2> 2226362 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@5ef4d123{HTTP/1.1,[http/1.1]}{127.0.0.1:42297}
   [junit4]   2> 2226362 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.e.j.s.Server Started @2226464ms
   [junit4]   2> 2226362 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest_24FFF8DB5625604C-001/tempDir-001/jetty1,
 replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=42297, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest_24FFF8DB5625604C-001/shard-1-001/cores}
   [junit4]   2> 2226363 ERROR (closeThreadPool-10362-thread-1) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 2226363 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter Using logger factory 
org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2226363 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
7.8.0
   [junit4]   2> 2226363 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2226363 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2226363 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2019-01-30T12:01:29.612Z
   [junit4]   2> 2226365 INFO  (zkConnectionManagerCallback-10364-thread-1) [   
 ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2226366 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 2226366 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest_24FFF8DB5625604C-001/shard-1-001/solr.xml
   [junit4]   2> 2226371 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay 
is ignored
   [junit4]   2> 2226371 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.c.SolrXmlConfig Configuration parameter 
autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2226373 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.c.SolrXmlConfig MBean server found: 
com.sun.jmx.mbeanserver.JmxMBeanServer@53e0c7af, but no JMX reporters were 
configured - adding default JMX reporter.
   [junit4]   2> 2226540 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest_24FFF8DB5625604C-001/shard-2-001
 of type NRT
   [junit4]   2> 2226542 WARN  (closeThreadPool-10362-thread-2) [    ] 
o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2226542 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2226542 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2226542 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: 
c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2226544 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2226544 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2226544 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2226545 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@60e4711b{/,null,AVAILABLE}
   [junit4]   2> 2226545 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@5c208096{HTTP/1.1,[http/1.1]}{127.0.0.1:34545}
   [junit4]   2> 2226545 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.e.j.s.Server Started @2226647ms
   [junit4]   2> 2226545 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest_24FFF8DB5625604C-001/tempDir-001/jetty2,
 replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=34545, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest_24FFF8DB5625604C-001/shard-2-001/cores}
   [junit4]   2> 2226546 ERROR (closeThreadPool-10362-thread-2) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 2226546 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter Using logger factory 
org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2226546 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
7.8.0
   [junit4]   2> 2226546 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2226546 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2226546 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2019-01-30T12:01:29.795Z
   [junit4]   2> 2226549 INFO  (zkConnectionManagerCallback-10366-thread-1) [   
 ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2226551 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 2226551 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest_24FFF8DB5625604C-001/shard-2-001/solr.xml
   [junit4]   2> 2226556 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay 
is ignored
   [junit4]   2> 2226556 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.a.s.c.SolrXmlConfig Configuration parameter 
autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2226558 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.a.s.c.SolrXmlConfig MBean server found: 
com.sun.jmx.mbeanserver.JmxMBeanServer@53e0c7af, but no JMX reporters were 
configured - adding default JMX reporter.
   [junit4]   2> 2226600 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: 
WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 2226606 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34379/solr
   [junit4]   2> 2226610 INFO  (zkConnectionManagerCallback-10370-thread-1) [   
 ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2226616 INFO  (zkConnectionManagerCallback-10372-thread-1) [   
 ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2226625 INFO  (closeThreadPool-10362-thread-1) 
[n:127.0.0.1:42297_    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 2226629 INFO  (closeThreadPool-10362-thread-1) 
[n:127.0.0.1:42297_    ] o.a.s.c.ZkController Publish node=127.0.0.1:42297_ as 
DOWN
   [junit4]   2> 2226631 INFO  (closeThreadPool-10362-thread-1) 
[n:127.0.0.1:42297_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating 
transient cache for 4 transient cores
   [junit4]   2> 2226631 INFO  (closeThreadPool-10362-thread-1) 
[n:127.0.0.1:42297_    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:42297_
   [junit4]   2> 2226632 INFO  (zkCallback-10342-thread-4) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2226632 INFO  (zkCallback-10349-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2226634 INFO  (zkCallback-10371-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2226639 INFO  (zkCallback-10360-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2226665 INFO  (zkConnectionManagerCallback-10379-thread-1) [   
 ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2226668 INFO  (closeThreadPool-10362-thread-1) 
[n:127.0.0.1:42297_    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (2)
   [junit4]   2> 2226670 INFO  (closeThreadPool-10362-thread-1) 
[n:127.0.0.1:42297_    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 
127.0.0.1:34379/solr ready
   [junit4]   2> 2226671 INFO  (closeThreadPool-10362-thread-1) 
[n:127.0.0.1:42297_    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, 
keeping metrics history in memory.
   [junit4]   2> 2226707 INFO  (closeThreadPool-10362-thread-1) 
[n:127.0.0.1:42297_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@53e0c7af
   [junit4]   2> 2226724 INFO  (closeThreadPool-10362-thread-1) 
[n:127.0.0.1:42297_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@53e0c7af
   [junit4]   2> 2226724 INFO  (closeThreadPool-10362-thread-1) 
[n:127.0.0.1:42297_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@53e0c7af
   [junit4]   2> 2226726 INFO  (closeThreadPool-10362-thread-1) 
[n:127.0.0.1:42297_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions 
underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest_24FFF8DB5625604C-001/shard-1-001/cores
   [junit4]   2> 2226735 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest_24FFF8DB5625604C-001/shard-3-001
 of type NRT
   [junit4]   2> 2226755 WARN  (closeThreadPool-10362-thread-3) [    ] 
o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2226756 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2226756 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2226756 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: 
c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2226758 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2226758 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2226758 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2226759 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@490f819{/,null,AVAILABLE}
   [junit4]   2> 2226760 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@1b27cdc2{HTTP/1.1,[http/1.1]}{127.0.0.1:36858}
   [junit4]   2> 2226760 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.e.j.s.Server Started @2226862ms
   [junit4]   2> 2226760 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest_24FFF8DB5625604C-001/tempDir-001/jetty3,
 replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=36858, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest_24FFF8DB5625604C-001/shard-3-001/cores}
   [junit4]   2> 2226761 ERROR (closeThreadPool-10362-thread-3) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 2226761 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter Using logger factory 
org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2226761 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
7.8.0
   [junit4]   2> 2226761 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2226761 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2226761 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2019-01-30T12:01:30.010Z
   [junit4]   2> 2226769 INFO  (zkConnectionManagerCallback-10382-thread-1) [   
 ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2226782 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 2226782 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest_24FFF8DB5625604C-001/shard-3-001/solr.xml
   [junit4]   2> 2226787 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay 
is ignored
   [junit4]   2> 2226787 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.a.s.c.SolrXmlConfig Configuration parameter 
autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2226789 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.a.s.c.SolrXmlConfig MBean server found: 
com.sun.jmx.mbeanserver.JmxMBeanServer@53e0c7af, but no JMX reporters were 
configured - adding default JMX reporter.
   [junit4]   2> 2226808 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: 
WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 2226844 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34379/solr
   [junit4]   2> 2226846 INFO  (zkConnectionManagerCallback-10386-thread-1) [   
 ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2226867 INFO  (zkConnectionManagerCallback-10388-thread-1) [   
 ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2226872 INFO  (closeThreadPool-10362-thread-2) 
[n:127.0.0.1:34545_    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (2)
   [junit4]   2> 2226876 INFO  (closeThreadPool-10362-thread-2) 
[n:127.0.0.1:34545_    ] o.a.s.c.ZkController Publish node=127.0.0.1:34545_ as 
DOWN
   [junit4]   2> 2226877 INFO  (closeThreadPool-10362-thread-2) 
[n:127.0.0.1:34545_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating 
transient cache for 4 transient cores
   [junit4]   2> 2226877 INFO  (closeThreadPool-10362-thread-2) 
[n:127.0.0.1:34545_    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:34545_
   [junit4]   2> 2226879 INFO  (zkCallback-10342-thread-4) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2226879 INFO  (zkCallback-10349-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2226879 INFO  (zkCallback-10360-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2226879 INFO  (zkCallback-10378-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2226879 INFO  (zkCallback-10387-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2226880 INFO  (zkCallback-10371-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2226931 INFO  (zkConnectionManagerCallback-10395-thread-1) [   
 ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2226934 INFO  (closeThreadPool-10362-thread-2) 
[n:127.0.0.1:34545_    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (3)
   [junit4]   2> 2226936 INFO  (closeThreadPool-10362-thread-2) 
[n:127.0.0.1:34545_    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 
127.0.0.1:34379/solr ready
   [junit4]   2> 2226937 INFO  (closeThreadPool-10362-thread-2) 
[n:127.0.0.1:34545_    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, 
keeping metrics history in memory.
   [junit4]   2> 2226984 INFO  (closeThreadPool-10362-thread-2) 
[n:127.0.0.1:34545_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@53e0c7af
   [junit4]   2> 2227009 INFO  (closeThreadPool-10362-thread-2) 
[n:127.0.0.1:34545_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@53e0c7af
   [junit4]   2> 2227009 INFO  (closeThreadPool-10362-thread-2) 
[n:127.0.0.1:34545_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@53e0c7af
   [junit4]   2> 2227013 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: 
WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 2227013 INFO  (closeThreadPool-10362-thread-2) 
[n:127.0.0.1:34545_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions 
underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest_24FFF8DB5625604C-001/shard-2-001/cores
   [junit4]   2> 2227018 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34379/solr
   [junit4]   2> 2227021 INFO  (zkConnectionManagerCallback-10400-thread-1) [   
 ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2227024 INFO  (zkConnectionManagerCallback-10402-thread-1) [   
 ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2227033 INFO  (closeThreadPool-10362-thread-3) 
[n:127.0.0.1:36858_    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (3)
   [junit4]   2> 2227041 INFO  (closeThreadPool-10362-thread-3) 
[n:127.0.0.1:36858_    ] o.a.s.c.ZkController Publish node=127.0.0.1:36858_ as 
DOWN
   [junit4]   2> 2227042 INFO  (closeThreadPool-10362-thread-3) 
[n:127.0.0.1:36858_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating 
transient cache for 4 transient cores
   [junit4]   2> 2227042 INFO  (closeThreadPool-10362-thread-3) 
[n:127.0.0.1:36858_    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:36858_
   [junit4]   2> 2227045 INFO  (zkCallback-10360-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2227045 INFO  (zkCallback-10342-thread-4) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2227045 INFO  (zkCallback-10387-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2227045 INFO  (zkCallback-10378-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2227045 INFO  (zkCallback-10349-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2227051 INFO  (zkCallback-10394-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2227051 INFO  (zkCallback-10371-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2227052 INFO  (zkCallback-10401-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2227079 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.c.AbstractFullDistribZkTestBase create jetty 4 in directory 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest_24FFF8DB5625604C-001/shard-4-001
 of type NRT
   [junit4]   2> 2227080 WARN  (closeThreadPool-10362-thread-1) [    ] 
o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2227080 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2227080 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2227080 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: 
c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2227091 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2227091 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2227091 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 2227091 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@3fdf54f0{/,null,AVAILABLE}
   [junit4]   2> 2227092 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@11e0a38{HTTP/1.1,[http/1.1]}{127.0.0.1:36327}
   [junit4]   2> 2227092 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.e.j.s.Server Started @2227194ms
   [junit4]   2> 2227092 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest_24FFF8DB5625604C-001/tempDir-001/jetty4,
 solrconfig=solrconfig.xml, hostContext=/, hostPort=36327, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest_24FFF8DB5625604C-001/shard-4-001/cores}
   [junit4]   2> 2227092 ERROR (closeThreadPool-10362-thread-1) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 2227092 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter Using logger factory 
org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2227092 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
7.8.0
   [junit4]   2> 2227092 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2227093 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2227093 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2019-01-30T12:01:30.342Z
   [junit4]   2> 2227111 INFO  (zkConnectionManagerCallback-10406-thread-1) [   
 ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2227113 INFO  (zkConnectionManagerCallback-10411-thread-1) [   
 ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2227114 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 2227114 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest_24FFF8DB5625604C-001/shard-4-001/solr.xml
   [junit4]   2> 2227115 INFO  (closeThreadPool-10362-thread-3) 
[n:127.0.0.1:36858_    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (4)
   [junit4]   2> 2227116 INFO  (closeThreadPool-10362-thread-3) 
[n:127.0.0.1:36858_    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 
127.0.0.1:34379/solr ready
   [junit4]   2> 2227116 INFO  (closeThreadPool-10362-thread-3) 
[n:127.0.0.1:36858_    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, 
keeping metrics history in memory.
   [junit4]   2> 2227120 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay 
is ignored
   [junit4]   2> 2227120 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.c.SolrXmlConfig Configuration parameter 
autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2227122 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.c.SolrXmlConfig MBean server found: 
com.sun.jmx.mbeanserver.JmxMBeanServer@53e0c7af, but no JMX reporters were 
configured - adding default JMX reporter.
   [junit4]   2> 2227171 INFO  (closeThreadPool-10362-thread-3) 
[n:127.0.0.1:36858_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@53e0c7af
   [junit4]   2> 2227207 INFO  (closeThreadPool-10362-thread-3) 
[n:127.0.0.1:36858_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@53e0c7af
   [junit4]   2> 2227208 INFO  (closeThreadPool-10362-thread-3) 
[n:127.0.0.1:36858_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@53e0c7af
   [junit4]   2> 2227211 INFO  (closeThreadPool-10362-thread-3) 
[n:127.0.0.1:36858_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions 
underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest_24FFF8DB5625604C-001/shard-3-001/cores
   [junit4]   2> 2227335 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.c.AbstractFullDistribZkTestBase create jetty 5 in directory 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest_24FFF8DB5625604C-001/shard-5-001
 of type NRT
   [junit4]   2> 2227336 WARN  (closeThreadPool-10362-thread-2) [    ] 
o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2227339 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2227339 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2227339 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: 
c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2227370 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2227370 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2227370 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2227371 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@5a9fb6d1{/,null,AVAILABLE}
   [junit4]   2> 2227372 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@6a6aecd{HTTP/1.1,[http/1.1]}{127.0.0.1:46855}
   [junit4]   2> 2227372 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.e.j.s.Server Started @2227474ms
   [junit4]   2> 2227372 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest_24FFF8DB5625604C-001/tempDir-001/jetty5,
 solrconfig=solrconfig.xml, hostContext=/, hostPort=46855, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest_24FFF8DB5625604C-001/shard-5-001/cores}
   [junit4]   2> 2227372 ERROR (closeThreadPool-10362-thread-2) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 2227372 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter Using logger factory 
org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2227372 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
7.8.0
   [junit4]   2> 2227373 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2227373 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2227373 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2019-01-30T12:01:30.622Z
   [junit4]   2> 2227395 INFO  (zkConnectionManagerCallback-10414-thread-1) [   
 ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2227396 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 2227396 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest_24FFF8DB5625604C-001/shard-5-001/solr.xml
   [junit4]   2> 2227397 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: 
WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 2227401 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay 
is ignored
   [junit4]   2> 2227401 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.a.s.c.SolrXmlConfig Configuration parameter 
autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2227402 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34379/solr
   [junit4]   2> 2227404 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.a.s.c.SolrXmlConfig MBean server found: 
com.sun.jmx.mbeanserver.JmxMBeanServer@53e0c7af, but no JMX reporters were 
configured - adding default JMX reporter.
   [junit4]   2> 2227404 INFO  (zkConnectionManagerCallback-10418-thread-1) [   
 ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2227440 INFO  (zkConnectionManagerCallback-10420-thread-1) [   
 ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2227446 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 2227454 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.c.ZkController Publish node=127.0.0.1:36327_ as DOWN
   [junit4]   2> 2227456 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 
transient cores
   [junit4]   2> 2227456 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:36327_
   [junit4]   2> 2227464 INFO  (zkCallback-10387-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2227464 INFO  (zkCallback-10360-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2227464 INFO  (zkCallback-10378-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2227464 INFO  (zkCallback-10342-thread-4) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2227465 INFO  (zkCallback-10371-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2227465 INFO  (zkCallback-10349-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2227465 INFO  (zkCallback-10401-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2227471 INFO  (zkCallback-10410-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2227472 INFO  (zkCallback-10394-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2227486 INFO  (zkCallback-10419-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2227535 INFO  (zkConnectionManagerCallback-10427-thread-1) [   
 ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2227537 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (5)
   [junit4]   2> 2227539 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34379/solr ready
   [junit4]   2> 2227540 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history 
in memory.
   [junit4]   2> 2227576 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') 
enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@53e0c7af
   [junit4]   2> 2227592 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[24FFF8DB5625604C]) [    ] 
o.a.s.c.AbstractFullDistribZkTestBase create jetty 6 in directory 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest_24FFF8DB5625604C-001/shard-6-001
 of type NRT
   [junit4]   2> 2227592 WARN  (closeThreadPool-10362-thread-3) [    ] 
o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2227593 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2227593 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2227593 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: 
c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2227596 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') 
enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@53e0c7af
   [junit4]   2> 2227596 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 
'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@53e0c7af
   [junit4]   2> 2227599 INFO  (closeThreadPool-10362-thread-1) [    ] 
o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest_24FFF8DB5625604C-001/shard-4-001/cores
   [junit4]   2> 2227600 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2227600 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2227600 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2227600 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@4eb3402e{/,null,AVAILABLE}
   [junit4]   2> 2227601 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@56254152{HTTP/1.1,[http/1.1]}{127.0.0.1:40594}
   [junit4]   2> 2227601 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.e.j.s.Server Started @2227703ms
   [junit4]   2> 2227601 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest_24FFF8DB5625604C-001/tempDir-001/jetty6,
 solrconfig=solrconfig.xml, hostContext=/, hostPort=40594, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest_24FFF8DB5625604C-001/shard-6-001/cores}
   [junit4]   2> 2227601 ERROR (closeThreadPool-10362-thread-3) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 2227601 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter Using logger factory 
org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2227601 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
7.8.0
   [junit4]   2> 2227601 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2227602 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2227602 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2019-01-30T12:01:30.851Z
   [junit4]   2> 2227604 INFO  (zkConnectionManagerCallback-10430-thread-1) [   
 ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2227605 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 2227605 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest_24FFF8DB5625604C-001/shard-6-001/solr.xml
   [junit4]   2> 2227610 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay 
is ignored
   [junit4]   2> 2227610 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.a.s.c.SolrXmlConfig Configuration parameter 
autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2227612 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.a.s.c.SolrXmlConfig MBean server found: 
com.sun.jmx.mbeanserver.JmxMBeanServer@53e0c7af, but no JMX reporters were 
configured - adding default JMX reporter.
   [junit4]   2> 2227749 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: 
WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 2227753 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34379/solr
   [junit4]   2> 2227754 INFO  (zkConnectionManagerCallback-10434-thread-1) [   
 ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2227757 INFO  (zkConnectionManagerCallback-10436-thread-1) [   
 ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2227762 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (5)
   [junit4]   2> 2227766 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.a.s.c.ZkController Publish node=127.0.0.1:46855_ as DOWN
   [junit4]   2> 2227767 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 
transient cores
   [junit4]   2> 2227767 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:46855_
   [junit4]   2> 2227769 INFO  (zkCallback-10410-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 2227769 INFO  (zkCallback-10378-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 2227769 INFO  (zkCallback-10394-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 2227769 INFO  (zkCallback-10342-thread-4) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 2227769 INFO  (zkCallback-10360-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 2227770 INFO  (zkCallback-10387-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 2227770 INFO  (zkCallback-10371-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 2227770 INFO  (zkCallback-10401-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 2227770 INFO  (zkCallback-10349-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 2227770 INFO  (zkCallback-10419-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 2227770 INFO  (zkCallback-10435-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 2227771 INFO  (zkCallback-10426-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 2227825 INFO  (zkConnectionManagerCallback-10443-thread-1) [   
 ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2227826 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (6)
   [junit4]   2> 2227827 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: 
WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 2227827 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34379/solr ready
   [junit4]   2> 2227828 INFO  (closeThreadPool-10362-thread-2) [    ] 
o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history 
in memory.
   [junit4]   2> 2227835 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34379/solr
   [junit4]   2> 2227837 INFO  (zkConnectionManagerCallback-10448-thread-1) [   
 ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2227840 INFO  (zkConnectionManagerCallback-10450-thread-1) [   
 ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2227846 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (6)
   [junit4]   2> 2227851 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.a.s.c.ZkController Publish node=127.0.0.1:40594_ as DOWN
   [junit4]   2> 2227852 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 
transient cores
   [junit4]   2> 2227852 INFO  (closeThreadPool-10362-thread-3) [    ] 
o.a.s.c.ZkController Register node as live in ZooKeeper:/li

[...truncated too long message...]

 [?:1.8.0_191]
   [junit4]   2> 5061219 ERROR 
(SUITE-HdfsRestartWhileUpdatingTest-seed#[24FFF8DB5625604C]-worker) [    ] 
o.a.s.c.SolrCore Too many close [count:-1] on 
org.apache.solr.core.SolrCore@48448da1. Please report this exception to 
solr-u...@lucene.apache.org
   [junit4]   2> 5061219 ERROR 
(SUITE-HdfsRestartWhileUpdatingTest-seed#[24FFF8DB5625604C]-worker) [    ] 
o.a.s.c.u.ObjectReleaseTracker 
   [junit4]   2> java.lang.AssertionError: Too many closes on SolrCore
   [junit4]   2>        at 
org.apache.solr.core.SolrCore.close(SolrCore.java:1525) ~[java/:?]
   [junit4]   2>        at 
org.apache.solr.common.util.ObjectReleaseTracker.tryClose(ObjectReleaseTracker.java:85)
 [java/:?]
   [junit4]   2>        at 
org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:322) 
[java/:?]
   [junit4]   2>        at 
sun.reflect.GeneratedMethodAccessor169.invoke(Unknown Source) ~[?:?]
   [junit4]   2>        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:1.8.0_191]
   [junit4]   2>        at java.lang.reflect.Method.invoke(Method.java:498) 
~[?:1.8.0_191]
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
 [randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:901)
 [randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 [randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
 [randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2>        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
 [java/:?]
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 [randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
 [java/:?]
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
 [randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
 [randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 [randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 [randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 [randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
 [java/:?]
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
 [java/:?]
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
 [java/:?]
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
 [java/:?]
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 [randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
 [randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsRestartWhileUpdatingTest_24FFF8DB5625604C-001
   [junit4]   2> Jan 30, 2019 12:48:44 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 32 leaked 
thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): 
{rnd_b=BlockTreeOrds(blocksize=128), 
a_t=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))),
 id=BlockTreeOrds(blocksize=128)}, 
docValues:{_version_=DocValuesFormat(name=Memory)}, maxPointsInLeafNode=1143, 
maxMBSortInHeap=7.422027716592088, sim=RandomSimilarity(queryNorm=true): {}, 
locale=en-MT, timezone=US/Arizona
   [junit4]   2> NOTE: Linux 4.4.0-137-generic amd64/Oracle Corporation 
1.8.0_191 (64-bit)/cpus=4,threads=4,free=168219360,total=511705088
   [junit4]   2> NOTE: All tests run in this JVM: 
[ProtectedTermFilterFactoryTest, DirectUpdateHandlerTest, 
TestBlendedInfixSuggestions, DistribJoinFromCollectionTest, TestComponentsName, 
EnumFieldTest, PreAnalyzedUpdateProcessorTest, TestTolerantSearch, 
TestPseudoReturnFields, ExecutePlanActionTest, MultiSolrCloudTestCaseTest, 
HighlighterTest, TestFieldSortValues, ShufflingReplicaListTransformerTest, 
SpellCheckCollatorTest, TestSha256AuthenticationProvider, CoreAdminHandlerTest, 
HdfsBasicDistributedZkTest, TestAuthorizationFramework, 
TestUniqueKeyFieldResource, HdfsDirectoryTest, TestManagedResourceStorage, 
TestFieldCacheWithThreads, TestConfigSets, ResponseHeaderTest, 
SubstringBytesRefFilterTest, TestCorePropertiesReload, TestSearchPerf, 
TestFieldTypeResource, TestLegacyFieldReuse, DirectSolrConnectionTest, 
TestLMDirichletSimilarityFactory, BigEndianAscendingWordDeserializerTest, 
SharedFSAutoReplicaFailoverTest, TestHdfsUpdateLog, TestLRUCache, 
LukeRequestHandlerTest, SearchRateTriggerIntegrationTest, 
PrimitiveFieldTypeTest, CdcrReplicationHandlerTest, TestNumericTerms64, 
TestIndexSearcher, CoreMergeIndexesAdminHandlerTest, 
TestDocTermOrdsUninvertLimit, SuggesterFSTTest, JsonLoaderTest, 
SimplePostToolTest, LIRRollingUpdatesTest, NotRequiredUniqueKeyTest, 
BasicAuthIntegrationTest, TestSolrQueryParser, MergeStrategyTest, 
TestMultiValuedNumericRangeQuery, NumberUtilsTest, 
DistributedFacetExistsSmallTest, ShardSplitTest, 
TestPHPSerializedResponseWriter, TestSolrCloudWithHadoopAuthPlugin, 
ZkSolrClientTest, DistributedTermsComponentTest, IndexSchemaTest, 
DeleteNodeTest, FastVectorHighlighterTest, SyncSliceTest, 
NodeAddedTriggerIntegrationTest, AnalysisErrorHandlingTest, 
HdfsTlogReplayBufferedWhileIndexingTest, BooleanFieldTest, TestPrepRecovery, 
TestMissingGroups, ZkFailoverTest, BasicFunctionalityTest, 
TestScoreJoinQPScore, TriggerIntegrationTest, OverseerModifyCollectionTest, 
CurrencyRangeFacetCloudTest, TestShardHandlerFactory, 
DistributedQueryElevationComponentTest, TestSolrConfigHandlerConcurrent, 
DeleteStatusTest, TaggerTest, OutOfBoxZkACLAndCredentialsProvidersTest, 
RankQueryTest, TestXmlQParserPlugin, TestRTGBase, 
AutoAddReplicasPlanActionTest, TestBadConfig, CollectionTooManyReplicasTest, 
LeaderElectionTest, EmbeddedSolrNoSerializeTest, TestConfigSetsAPIExclusivity, 
ClassificationUpdateProcessorTest, TestCopyFieldCollectionResource, 
TestCloudNestedDocsSort, TestGeoJSONResponseWriter, TestTlogReplica, 
DateMathParserTest, SolrGraphiteReporterTest, BasicDistributedZkTest, 
TestRandomFaceting, ConcurrentDeleteAndCreateCollectionTest, PolyFieldTest, 
TestMaxTokenLenTokenizer, TestSimPolicyCloud, TestPartialUpdateDeduplication, 
IndexSizeTriggerTest, TestRequestForwarding, SolrSlf4jReporterTest, 
TestOverriddenPrefixQueryForCustomFieldType, ClusterStateTest, 
TestDistributedSearch, TransactionLogTest, TestReplicationHandlerBackup, 
TestMacros, V2StandaloneTest, CreateRoutedAliasTest, SolrInfoBeanTest, 
ActionThrottleTest, TestStressUserVersions, SynonymTokenizerTest, 
DebugComponentTest, TestStressCloudBlindAtomicUpdates, TestReplicaProperties, 
TestManagedSynonymGraphFilterFactory, TestDownShardTolerantSearch, 
DistributedMLTComponentTest, ExplicitHLLTest, TestUtilizeNode, 
DocValuesNotIndexedTest, StressHdfsTest, TestCoreBackup, UtilsToolTest, 
ConfigureRecoveryStrategyTest, TestLuceneIndexBackCompat, 
TestSolrCloudSnapshots, TestZkChroot, TlogReplayBufferedWhileIndexingTest, 
UnloadDistributedZkTest, HdfsCollectionsAPIDistributedZkTest, 
TestCollectionAPI, TestCollectionsAPIViaSolrCloudCluster, 
HttpTriggerListenerTest, NodeLostTriggerIntegrationTest, NodeLostTriggerTest, 
ScheduledMaintenanceTriggerTest, ScheduledTriggerIntegrationTest, 
TestPolicyCloud, TestSimNodeAddedTrigger, CdcrBidirectionalTest, 
CdcrBootstrapTest, HdfsRecoverLeaseTest, HdfsRestartWhileUpdatingTest]
   [junit4] Completed [576/838 (2!)] on J1 in 363.65s, 1 test, 1 failure <<< 
FAILURES!

[...truncated 901 lines...]
   [junit4] JVM J2: stdout was not empty, see: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/solr/build/solr-core/test/temp/junit4-J2-20190130_112423_0655367413650922641614.sysout
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/heapdumps/java_pid17990.hprof
 ...
   [junit4] Heap dump file created [718310965 bytes in 6.030 secs]
   [junit4] <<< JVM J2: EOF ----

[...truncated 8884 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/build.xml:661:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-7.x/checkout/build.xml:585:
 Some of the tests produced a heap dump, but did not fail. Maybe a suppressed 
OutOfMemoryError? Dumps created:
* java_pid17990.hprof

Total time: 287 minutes 37 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Reply via email to