Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/28/

9 tests failed.
FAILED:  org.apache.solr.cloud.CloudExitableDirectoryReaderTest.test

Error Message:
No live SolrServers available to handle this 
request:[https://127.0.0.1:38388/xi/j/collection1]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available 
to handle this request:[https://127.0.0.1:38388/xi/j/collection1]
        at 
__randomizedtesting.SeedInfo.seed([F4FEB0EA8AE0D98C:7CAA8F30241CB474]:0)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:352)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1165)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:935)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:871)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:957)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.queryServer(AbstractFullDistribZkTestBase.java:1397)
        at 
org.apache.solr.cloud.CloudExitableDirectoryReaderTest.assertPartialResults(CloudExitableDirectoryReaderTest.java:99)
        at 
org.apache.solr.cloud.CloudExitableDirectoryReaderTest.doTimeoutTests(CloudExitableDirectoryReaderTest.java:71)
        at 
org.apache.solr.cloud.CloudExitableDirectoryReaderTest.test(CloudExitableDirectoryReaderTest.java:50)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)
Caused by: 
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at https://127.0.0.1:38388/xi/j/collection1: 
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available 
to handle this request
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:577)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:372)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:325)
        ... 51 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestCloudDeleteByQuery

Error Message:
There are still nodes recoverying - waited for 330 seconds

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 330 
seconds
        at __randomizedtesting.SeedInfo.seed([F4FEB0EA8AE0D98C]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:173)
        at 
org.apache.solr.cloud.TestCloudDeleteByQuery.createMiniSolrCloudCluster(TestCloudDeleteByQuery.java:113)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:811)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  
org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.testVariousDeletesViaShard1NonLeaderClient

Error Message:
org.apache.solr.client.solrj.impl.CloudSolrClient@3c28b8b5 should find id: 
XYZ!666 expected:<1> but was:<0>

Stack Trace:
java.lang.AssertionError: 
org.apache.solr.client.solrj.impl.CloudSolrClient@3c28b8b5 should find id: 
XYZ!666 expected:<1> but was:<0>
        at 
__randomizedtesting.SeedInfo.seed([F4FEB0EA8AE0D98C:2F2EA7F1221C332B]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at 
org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.assertQueryDocIds(TestTolerantUpdateProcessorCloud.java:1002)
        at 
org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.assertQueryDocIds(TestTolerantUpdateProcessorCloud.java:1007)
        at 
org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.testVariousDeletes(TestTolerantUpdateProcessorCloud.java:319)
        at 
org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.testVariousDeletesViaShard1NonLeaderClient(TestTolerantUpdateProcessorCloud.java:278)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  
org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.testVariousAddsViaShard1NonLeaderClient

Error Message:
org.apache.solr.client.solrj.impl.CloudSolrClient@3c28b8b5 should find id: 
XYZ!666 expected:<1> but was:<0>

Stack Trace:
java.lang.AssertionError: 
org.apache.solr.client.solrj.impl.CloudSolrClient@3c28b8b5 should find id: 
XYZ!666 expected:<1> but was:<0>
        at 
__randomizedtesting.SeedInfo.seed([F4FEB0EA8AE0D98C:63C4E63748A7CBA]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at 
org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.assertQueryDocIds(TestTolerantUpdateProcessorCloud.java:1002)
        at 
org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.assertQueryDocIds(TestTolerantUpdateProcessorCloud.java:1007)
        at 
org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.testVariousAdds(TestTolerantUpdateProcessorCloud.java:450)
        at 
org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.testVariousAddsViaShard1NonLeaderClient(TestTolerantUpdateProcessorCloud.java:389)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  
org.apache.solr.cloud.TestTolerantUpdateProcessorRandomCloud.testRandomUpdates

Error Message:
org.apache.solr.client.solrj.impl.HttpSolrClient@54379d12 => 
[type=<DELQ>,id=<foo_i:[27643 TO ....giberish>,msgSubStr=<null>]: Num error 
ids: [{type=DELID,id=id_37373,message=No registered leader was found after 
waiting for 4000ms , collection: test_col slice: shard3}, 
{type=DELQ,id=foo_i:[27643 TO ....giberish,message=No registered leader was 
found after waiting for 4000ms , collection: test_col slice: shard3}] 
expected:<1> but was:<2>

Stack Trace:
java.lang.AssertionError: 
org.apache.solr.client.solrj.impl.HttpSolrClient@54379d12 => 
[type=<DELQ>,id=<foo_i:[27643 TO ....giberish>,msgSubStr=<null>]: Num error 
ids: [{type=DELID,id=id_37373,message=No registered leader was found after 
waiting for 4000ms , collection: test_col slice: shard3}, 
{type=DELQ,id=foo_i:[27643 TO ....giberish,message=No registered leader was 
found after waiting for 4000ms , collection: test_col slice: shard3}] 
expected:<1> but was:<2>
        at 
__randomizedtesting.SeedInfo.seed([F4FEB0EA8AE0D98C:C83F655AFD9D279E]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at 
org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.assertUpdateTolerantErrors(TestTolerantUpdateProcessorCloud.java:958)
        at 
org.apache.solr.cloud.TestTolerantUpdateProcessorRandomCloud.testRandomUpdates(TestTolerantUpdateProcessorRandomCloud.java:263)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest.test

Error Message:
There were too many update fails (79 > 60) - we expect it can happen, but 
shouldn't easily

Stack Trace:
java.lang.AssertionError: There were too many update fails (79 > 60) - we 
expect it can happen, but shouldn't easily
        at 
__randomizedtesting.SeedInfo.seed([F4FEB0EA8AE0D98C:7CAA8F30241CB474]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertFalse(Assert.java:68)
        at 
org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test(ChaosMonkeyNothingIsSafeTest.java:216)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  
org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy

Error Message:
Could not find collection : c1

Stack Trace:
org.apache.solr.common.SolrException: Could not find collection : c1
        at 
__randomizedtesting.SeedInfo.seed([F4FEB0EA8AE0D98C:9FB11097F3EF04B6]:0)
        at 
org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:170)
        at 
org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdate(ZkStateReaderTest.java:135)
        at 
org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy(ZkStateReaderTest.java:46)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test

Error Message:


Stack Trace:
java.lang.AssertionError
        at 
__randomizedtesting.SeedInfo.seed([F4FEB0EA8AE0D98C:7CAA8F30241CB474]:0)
        at org.junit.Assert.fail(Assert.java:92)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertTrue(Assert.java:54)
        at 
org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.testBasics(SharedFSAutoReplicaFailoverTest.java:226)
        at 
org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test(SharedFSAutoReplicaFailoverTest.java:107)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: 
http://127.0.0.1:48892/r_bve

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting 
response from server at: http://127.0.0.1:48892/r_bve
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:588)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:381)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:497)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:169)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:170)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at 
org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
        at 
org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
        at 
org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
        at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
        at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
        at 
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
        at 
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
        at 
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
        at 
org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
        at 
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
        at 
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
        at 
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
        at 
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
        at 
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:482)
        ... 47 more




Build Log:
[...truncated 11334 lines...]
   [junit4] Suite: org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest
   [junit4]   2> Creating dataDir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_F4FEB0EA8AE0D98C-001/init-core-data-001
   [junit4]   2> 302747 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[F4FEB0EA8AE0D98C]-worker) [    ] 
o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /nyq/y
   [junit4]   2> 305075 WARN  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[F4FEB0EA8AE0D98C]-worker) [    ] 
o.a.h.u.NativeCodeLoader Unable to load native-hadoop library for your 
platform... using builtin-java classes where applicable
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 307693 WARN  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[F4FEB0EA8AE0D98C]-worker) [    ] 
o.a.h.m.i.MetricsConfig Cannot locate configuration: tried 
hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 308257 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[F4FEB0EA8AE0D98C]-worker) [    ] 
o.m.log Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via 
org.mortbay.log.Slf4jLog
   [junit4]   2> 308264 WARN  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[F4FEB0EA8AE0D98C]-worker) [    ] 
o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 308426 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[F4FEB0EA8AE0D98C]-worker) [    ] 
o.m.log jetty-6.1.26
   [junit4]   2> 308535 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[F4FEB0EA8AE0D98C]-worker) [    ] 
o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/hdfs
 to ./temp/Jetty_localhost_57114_hdfs____.quy9t0/webapp
   [junit4]   2> 308756 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[F4FEB0EA8AE0D98C]-worker) [    ] 
o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 310287 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[F4FEB0EA8AE0D98C]-worker) [    ] 
o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:57114
   [junit4]   2> 313087 WARN  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[F4FEB0EA8AE0D98C]-worker) [    ] 
o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 313116 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[F4FEB0EA8AE0D98C]-worker) [    ] 
o.m.log jetty-6.1.26
   [junit4]   2> 313165 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[F4FEB0EA8AE0D98C]-worker) [    ] 
o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode
 to ./temp/Jetty_localhost_37532_datanode____lj0m6v/webapp
   [junit4]   2> 313623 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[F4FEB0EA8AE0D98C]-worker) [    ] 
o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 314750 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[F4FEB0EA8AE0D98C]-worker) [    ] 
o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37532
   [junit4]   2> 315208 WARN  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[F4FEB0EA8AE0D98C]-worker) [    ] 
o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 315213 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[F4FEB0EA8AE0D98C]-worker) [    ] 
o.m.log jetty-6.1.26
   [junit4]   2> 315272 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[F4FEB0EA8AE0D98C]-worker) [    ] 
o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode
 to ./temp/Jetty_localhost_33991_datanode____.wcfrag/webapp
   [junit4]   2> 315618 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[F4FEB0EA8AE0D98C]-worker) [    ] 
o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 316573 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[F4FEB0EA8AE0D98C]-worker) [    ] 
o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33991
   [junit4]   2> 319393 INFO  (IPC Server handler 0 on 49490) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-f2989bfa-d2be-46d4-801c-10a4533a06b6 node DatanodeRegistration(127.0.0.1, 
datanodeUuid=eac22cf2-64b4-443d-87e1-906b4fb2595c, infoPort=33991, 
ipcPort=43835, storageInfo=lv=-56;cid=testClusterID;nsid=1023663521;c=0), 
blocks: 0, hasStaleStorages: true, processing time: 19 msecs
   [junit4]   2> 319394 INFO  (IPC Server handler 0 on 49490) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-1277a29c-bae7-4c57-b662-fc0d797e601b node DatanodeRegistration(127.0.0.1, 
datanodeUuid=eac22cf2-64b4-443d-87e1-906b4fb2595c, infoPort=33991, 
ipcPort=43835, storageInfo=lv=-56;cid=testClusterID;nsid=1023663521;c=0), 
blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 319394 INFO  (IPC Server handler 9 on 49490) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-19ad2966-5883-4597-818a-bab8e5c7f705 node DatanodeRegistration(127.0.0.1, 
datanodeUuid=b18edb30-2702-42ea-bd9e-f04e8cb0f700, infoPort=37532, 
ipcPort=40472, storageInfo=lv=-56;cid=testClusterID;nsid=1023663521;c=0), 
blocks: 0, hasStaleStorages: true, processing time: 10 msecs
   [junit4]   2> 319433 INFO  (IPC Server handler 9 on 49490) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-d75f0301-8de2-4b9d-8d2e-aaf34945166e node DatanodeRegistration(127.0.0.1, 
datanodeUuid=b18edb30-2702-42ea-bd9e-f04e8cb0f700, infoPort=37532, 
ipcPort=40472, storageInfo=lv=-56;cid=testClusterID;nsid=1023663521;c=0), 
blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 320026 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 320033 INFO  (Thread-735) [    ] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 320033 INFO  (Thread-735) [    ] o.a.s.c.ZkTestServer Starting 
server
   [junit4]   2> 320132 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.ZkTestServer start zk server on port:32850
   [junit4]   2> 320132 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 320144 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 320185 INFO  (zkCallback-220-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@4e8789bc 
name:ZooKeeperConnection Watcher:127.0.0.1:32850 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 320186 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 320186 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 320186 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 320188 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 320202 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 320213 INFO  (zkCallback-221-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@52546d32 
name:ZooKeeperConnection Watcher:127.0.0.1:32850/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 320213 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 320213 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 320213 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 320215 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 320215 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 320216 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 320216 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
 to /configs/conf1/solrconfig.xml
   [junit4]   2> 320217 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 320218 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/schema.xml
 to /configs/conf1/schema.xml
   [junit4]   2> 320218 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 320221 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
 to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 320221 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.c.SolrZkClient makePath: 
/configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 320222 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
 to /configs/conf1/stopwords.txt
   [junit4]   2> 320222 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 320223 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt
 to /configs/conf1/protwords.txt
   [junit4]   2> 320223 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 320224 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/currency.xml
 to /configs/conf1/currency.xml
   [junit4]   2> 320224 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 320224 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
 to /configs/conf1/enumsConfig.xml
   [junit4]   2> 320224 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 320225 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
 to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 320225 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 320226 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
 to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 320226 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 320227 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
 to /configs/conf1/old_synonyms.txt
   [junit4]   2> 320227 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 320227 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
 to /configs/conf1/synonyms.txt
   [junit4]   2> 320227 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 320445 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.SolrTestCaseJ4 Writing core.properties file to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_F4FEB0EA8AE0D98C-001/control-001/cores/collection1
   [junit4]   2> 320454 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 320461 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@75078734{/nyq/y,null,AVAILABLE}
   [junit4]   2> 320462 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.e.j.s.ServerConnector Started 
ServerConnector@2ccd46b7{HTTP/1.1,[http/1.1]}{127.0.0.1:34687}
   [junit4]   2> 320462 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.e.j.s.Server Started @326947ms
   [junit4]   2> 320462 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/nyq/y, 
hostPort=34687, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_F4FEB0EA8AE0D98C-001/control-001/cores}
   [junit4]   2> 320462 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): 
sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 320462 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_F4FEB0EA8AE0D98C-001/control-001'
   [junit4]   2> 320462 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 320462 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find 
system property or JNDI)
   [junit4]   2> 320462 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 320463 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 320473 INFO  (zkCallback-222-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@f9918ea name:ZooKeeperConnection 
Watcher:127.0.0.1:32850/solr got event WatchedEvent state:SyncConnected 
type:None path:null path:null type:None
   [junit4]   2> 320473 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 320473 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 320475 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 320475 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_F4FEB0EA8AE0D98C-001/control-001/solr.xml
   [junit4]   2> 320479 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.CorePropertiesLocator Config-defined core root directory: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_F4FEB0EA8AE0D98C-001/control-001/cores
   [junit4]   2> 320479 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.CoreContainer New CoreContainer 2059644463
   [junit4]   2> 320479 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.CoreContainer Loading cores into CoreContainer 
[instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_F4FEB0EA8AE0D98C-001/control-001]
   [junit4]   2> 320479 WARN  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.CoreContainer Couldn't add files from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_F4FEB0EA8AE0D98C-001/control-001/lib
 to classpath: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_F4FEB0EA8AE0D98C-001/control-001/lib
   [junit4]   2> 320479 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme 
: ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 
10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 
5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 320480 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 320480 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 320480 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.l.LogWatcher Registering Log Listener [Log4j 
(org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 320481 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:32850/solr
   [junit4]   2> 320481 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 320481 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 320486 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 320488 INFO  (zkCallback-225-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@1f5ba90f 
name:ZooKeeperConnection Watcher:127.0.0.1:32850 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 320489 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 320489 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 320555 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.ConnectionManager Waiting for client 
to connect to ZooKeeper
   [junit4]   2> 320589 INFO  
(zkCallback-226-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@904a049 name:ZooKeeperConnection 
Watcher:127.0.0.1:32850/solr got event WatchedEvent state:SyncConnected 
type:None path:null path:null type:None
   [junit4]   2> 320589 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.ConnectionManager Client is connected 
to ZooKeeper
   [junit4]   2> 320597 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 320640 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-queue-work
   [junit4]   2> 320705 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-map-running
   [junit4]   2> 320765 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-map-completed
   [junit4]   2> 320797 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-map-failure
   [junit4]   2> 320833 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 320835 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 320835 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.SolrZkClient makePath: 
/clusterstate.json
   [junit4]   2> 320836 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 320836 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.ZkStateReader Updating cluster state 
from ZooKeeper... 
   [junit4]   2> 320849 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (0)
   [junit4]   2> 320877 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 320878 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer_elect/election
   [junit4]   2> 320879 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 320880 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.LeaderElector Joined leadership 
election with path: 
/overseer_elect/election/95660805454299140-127.0.0.1:34687_nyq%2Fy-n_0000000000
   [junit4]   2> 320880 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.OverseerElectionContext I am going to 
be the leader 127.0.0.1:34687_nyq%2Fy
   [junit4]   2> 320880 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer_elect/leader
   [junit4]   2> 320881 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.Overseer Overseer 
(id=95660805454299140-127.0.0.1:34687_nyq%2Fy-n_0000000000) starting
   [junit4]   2> 320883 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/queue-work
   [junit4]   2> 320886 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.OverseerAutoReplicaFailoverThread 
Starting OverseerAutoReplicaFailoverThread 
autoReplicaFailoverWorkLoopDelay=10000 
autoReplicaFailoverWaitAfterExpiration=10000 
autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 320887 INFO  
(OverseerCollectionConfigSetProcessor-95660805454299140-127.0.0.1:34687_nyq%2Fy-n_0000000000)
 [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.OverseerTaskProcessor Process current 
queue of overseer operations
   [junit4]   2> 320887 INFO  
(OverseerStateUpdate-95660805454299140-127.0.0.1:34687_nyq%2Fy-n_0000000000) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.Overseer Starting to work on the main 
queue
   [junit4]   2> 320887 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:34687_nyq%2Fy
   [junit4]   2> 320889 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.SolrZkClient makePath: 
/live_nodes/127.0.0.1:34687_nyq%2Fy
   [junit4]   2> 320895 INFO  
(zkCallback-226-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.ZkStateReader A live node change: 
[WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], 
has occurred - updating... (live nodes size: [0])
   [junit4]   2> 320901 INFO  
(OverseerStateUpdate-95660805454299140-127.0.0.1:34687_nyq%2Fy-n_0000000000) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 320901 INFO  
(zkCallback-226-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (1) -> (1)
   [junit4]   2> 320909 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.CoreContainer Security conf doesn't 
exist. Skipping setup for authorization module.
   [junit4]   2> 320909 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.CoreContainer No authentication plugin 
used.
   [junit4]   2> 320913 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.CorePropertiesLocator Looking for core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_F4FEB0EA8AE0D98C-001/control-001/cores
   [junit4]   2> 320924 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: 
{name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, 
loadOnStartup=true, configSetProperties=configsetprops.json, 
collection=control_collection, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 320924 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.CorePropertiesLocator Found core 
collection1 in 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_F4FEB0EA8AE0D98C-001/control-001/cores/collection1
   [junit4]   2> 320924 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.CorePropertiesLocator Found 1 core 
definitions
   [junit4]   2> 320925 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] 
o.a.s.c.ZkController publishing state=down
   [junit4]   2> 320925 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] 
o.a.s.c.ZkController numShards not found on descriptor - reading it from system 
property
   [junit4]   2> 320928 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] 
o.a.s.c.ZkController look for our core node name
   [junit4]   2> 320936 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.s.SolrDispatchFilter 
user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2
   [junit4]   2> 320936 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.s.SolrDispatchFilter 
SolrDispatchFilter.init() done
   [junit4]   2> 320937 INFO  
(OverseerStateUpdate-95660805454299140-127.0.0.1:34687_nyq%2Fy-n_0000000000) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.Overseer processMessage: queueSize: 1, 
message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:34687/nyq/y";,
   [junit4]   2>   "node_name":"127.0.0.1:34687_nyq%2Fy",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 320937 INFO  
(OverseerStateUpdate-95660805454299140-127.0.0.1:34687_nyq%2Fy-n_0000000000) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.o.ReplicaMutator Update state 
numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:34687/nyq/y";,
   [junit4]   2>   "node_name":"127.0.0.1:34687_nyq%2Fy",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 320938 INFO  
(OverseerStateUpdate-95660805454299140-127.0.0.1:34687_nyq%2Fy-n_0000000000) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.o.ClusterStateMutator building a new 
cName: control_collection
   [junit4]   2> 320938 INFO  
(OverseerStateUpdate-95660805454299140-127.0.0.1:34687_nyq%2Fy-n_0000000000) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.o.ReplicaMutator Assigning new node to 
shard shard=shard1
   [junit4]   2> 320939 INFO  
(zkCallback-226-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 321929 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] 
o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 321929 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] 
o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 321929 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] 
o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 321929 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] 
o.a.s.c.c.ZkStateReader Load collection config from: 
[/collections/control_collection]
   [junit4]   2> 321930 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] 
o.a.s.c.c.ZkStateReader path=[/collections/control_collection] 
[configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 321930 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] 
o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_F4FEB0EA8AE0D98C-001/control-001/cores/collection1'
   [junit4]   2> 321930 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] 
o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 321930 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] 
o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find 
system property or JNDI)
   [junit4]   2> 321931 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] o.a.s.c.Config 
loaded config solrconfig.xml with version 0 
   [junit4]   2> 321982 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] 
o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 321993 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.1.0
   [junit4]   2> 322012 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] 
o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 322037 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] 
o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 322576 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Initialized with 
rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 322596 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] 
o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 322611 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] 
o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 322633 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] 
o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 322635 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] 
o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 322637 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from 
open-exchange-rates.json
   [junit4]   2> 322637 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from 
open-exchange-rates.json
   [junit4]   2> 322646 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] 
o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default 
properties: Can't find resource 'configsetprops.json' in classpath or 
'/configs/conf1', 
cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2
   [junit4]   2> 322646 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] 
o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from 
collection control_collection
   [junit4]   2> 322646 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore org.apache.solr.core.HdfsDirectoryFactory
   [junit4]   2> 322677 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.HdfsDirectoryFactory 
solr.hdfs.home=hdfs://localhost:49490/solr_hdfs_home
   [junit4]   2> 322677 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication 
disabled
   [junit4]   2> 322685 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_F4FEB0EA8AE0D98C-001/control-001/cores/collection1],
 
dataDir=[hdfs://localhost:49490/solr_hdfs_home/control_collection/core_node1/data/]
   [junit4]   2> 322685 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr 
mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@60576839
   [junit4]   2> 322686 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:49490/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 322756 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for 
hdfs://localhost:49490/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 322760 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore New index directory detected: old=null 
new=hdfs://localhost:49490/solr_hdfs_home/control_collection/core_node1/data/index/
   [junit4]   2> 322783 WARN  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory 
'hdfs:/localhost:49490/solr_hdfs_home/control_collection/core_node1/data/index' 
doesn't exist. Creating new index...
   [junit4]   2> 322783 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:49490/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 322789 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] 
with direct memory allocation set to [true]
   [junit4]   2> 322789 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, 
slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 322789 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS 
BlockCache
   [junit4]   2> 326084 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 326084 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for 
hdfs://localhost:49490/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 326084 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: 
minMergeSize=1000, mergeFactor=9, maxMergeSize=9223372036854775807, 
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, 
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=1.0]
   [junit4]   2> 326698 INFO  (IPC Server handler 5 on 49490) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:35579 is 
added to blk_1073741825_1001{blockUCState=COMMITTED, primaryNodeIndex=-1, 
replicas=[ReplicaUnderConstruction[[DISK]DS-d75f0301-8de2-4b9d-8d2e-aaf34945166e:NORMAL:127.0.0.1:49587|RBW],
 
ReplicaUnderConstruction[[DISK]DS-1277a29c-bae7-4c57-b662-fc0d797e601b:NORMAL:127.0.0.1:35579|RBW]]}
 size 71
   [junit4]   2> 326702 INFO  (IPC Server handler 1 on 49490) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49587 is 
added to blk_1073741825_1001 size 71
   [junit4]   2> 327134 INFO  
(coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) 
[n:127.0.0.1:34687_nyq%2Fy c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: 
num=1
   [junit4]   2>        
commit{dir=NRTCachingDirectory(BlockDirectory(HdfsDirectory@hdfs://localhost:49490/solr_hdfs_home/control_collection/core_node1/data/index
 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@54ccbd95); 
maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 327134 INFO  (c

[...truncated too long message...]

at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:214)
   [junit4]   2>        at 
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
   [junit4]   2>        at 
org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:266)
   [junit4]   2>        at 
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2>        at 
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 6386175 ERROR 
(zkCallback-2537-thread-10-processing-n:127.0.0.1:44804_r_bve-EventThread) 
[n:127.0.0.1:44804_r_bve    ] o.a.z.ClientCnxn Error while calling watcher 
   [junit4]   2> java.util.concurrent.RejectedExecutionException: Task 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$6/1370100541@4c2958b0
 rejected from 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@4f433467[Terminated,
 pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1773]
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:214)
   [junit4]   2>        at 
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
   [junit4]   2>        at 
org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:266)
   [junit4]   2>        at 
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2>        at 
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 6386175 ERROR 
(zkCallback-2537-thread-10-processing-n:127.0.0.1:44804_r_bve-EventThread) 
[n:127.0.0.1:44804_r_bve    ] o.a.z.ClientCnxn Error while calling watcher 
   [junit4]   2> java.util.concurrent.RejectedExecutionException: Task 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$6/1370100541@6c1db556
 rejected from 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@4f433467[Terminated,
 pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1773]
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:214)
   [junit4]   2>        at 
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
   [junit4]   2>        at 
org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:266)
   [junit4]   2>        at 
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2>        at 
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 6386175 ERROR 
(zkCallback-2537-thread-10-processing-n:127.0.0.1:44804_r_bve-EventThread) 
[n:127.0.0.1:44804_r_bve    ] o.a.z.ClientCnxn Error while calling watcher 
   [junit4]   2> java.util.concurrent.RejectedExecutionException: Task 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$6/1370100541@42e4e99b
 rejected from 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@4f433467[Terminated,
 pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1773]
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:214)
   [junit4]   2>        at 
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
   [junit4]   2>        at 
org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:266)
   [junit4]   2>        at 
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2>        at 
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene60): 
{other_tl1=PostingsFormat(name=LuceneVarGapDocFreqInterval), 
range_facet_l_dv=PostingsFormat(name=MockRandom), 
_version_=PostingsFormat(name=LuceneVarGapDocFreqInterval), 
multiDefault=PostingsFormat(name=Memory doPackFST= false), 
a_t=PostingsFormat(name=Memory doPackFST= false), 
intDefault=PostingsFormat(name=LuceneVarGapDocFreqInterval), 
a_i1=PostingsFormat(name=Memory doPackFST= false), 
id=PostingsFormat(name=MockRandom), range_facet_i_dv=PostingsFormat(name=Memory 
doPackFST= false), text=FST50, range_facet_l=PostingsFormat(name=Memory 
doPackFST= false), timestamp=PostingsFormat(name=Memory doPackFST= false)}, 
docValues:{range_facet_l_dv=DocValuesFormat(name=Lucene54), 
range_facet_i_dv=DocValuesFormat(name=Lucene54), 
timestamp=DocValuesFormat(name=Lucene54)}, maxPointsInLeafNode=77, 
maxMBSortInHeap=5.235541453874914, sim=ClassicSimilarity, locale=sk, 
timezone=Europe/Chisinau
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 
1.8.0_74 (64-bit)/cpus=4,threads=1,free=130955528,total=517996544
   [junit4]   2> NOTE: All tests run in this JVM: 
[TestPHPSerializedResponseWriter, TestReloadDeadlock, TestCollectionAPI, 
TestFuzzyAnalyzedSuggestions, DeleteLastCustomShardedReplicaTest, 
TestXIncludeConfig, TestConfigSetImmutable, AssignTest, CleanupOldIndexTest, 
AnalyticsMergeStrategyTest, HighlighterConfigTest, 
DistributedQueryComponentOptimizationTest, 
BigEndianAscendingWordSerializerTest, SearchHandlerTest, 
CoreMergeIndexesAdminHandlerTest, XmlUpdateRequestHandlerTest, 
ClusterStateTest, TestJsonRequest, ScriptEngineTest, AutoCommitTest, 
DirectUpdateHandlerTest, SharedFSAutoReplicaFailoverTest, 
DirectUpdateHandlerOptimizeTest, TestStressRecovery, 
TestSolrConfigHandlerConcurrent, TestXmlQParser, ReplicationFactorTest, 
OverseerCollectionConfigSetProcessorTest, TestSortingResponseWriter, 
TestClusterStateMutator, DistribJoinFromCollectionTest, 
TestExtendedDismaxParser, PKIAuthenticationIntegrationTest, 
TestDownShardTolerantSearch, UnloadDistributedZkTest, 
FullSolrCloudDistribCmdsTest, SolrTestCaseJ4Test, RequestHandlersTest, 
ForceLeaderTest, TestWriterPerf, TestBulkSchemaConcurrent, 
CollectionsAPISolrJTest, ClassificationUpdateProcessorFactoryTest, 
TestRequestStatusCollectionAPI, FieldAnalysisRequestHandlerTest, 
OverseerTaskQueueTest, DebugComponentTest, BlockCacheTest, TestSearchPerf, 
TestFieldTypeCollectionResource, DateMathParserTest, 
TestSolrCloudWithKerberosAlt, UpdateParamsTest, SolrCloudExampleTest, 
HdfsDirectoryFactoryTest, SegmentsInfoRequestHandlerTest, 
SolrCoreCheckLockOnStartupTest, TestOrdValues, TestDocBasedVersionConstraints, 
TlogReplayBufferedWhileIndexingTest, TestCopyFieldCollectionResource, 
TestDefaultStatsCache, TestIndexSearcher, 
TestOverriddenPrefixQueryForCustomFieldType, ConnectionManagerTest, 
CachingDirectoryFactoryTest, DocValuesMissingTest, StatsComponentTest, 
TestSchemaResource, BasicDistributedZk2Test, SampleTest, SparseHLLTest, 
BadCopyFieldTest, TestJmxIntegration, TestRecovery, HdfsDirectoryTest, 
TestCSVResponseWriter, IgnoreCommitOptimizeUpdateProcessorFactoryTest, 
TestDistributedStatsComponentCardinality, TestInitParams, TestFilteredDocIdSet, 
TestCollationField, TestRebalanceLeaders, HdfsSyncSliceTest, 
TermVectorComponentDistributedTest, TestSort, HardAutoCommitTest, 
TestFileDictionaryLookup, TestComplexPhraseQParserPlugin, RecoveryZkTest, 
NoCacheHeaderTest, TestRuleBasedAuthorizationPlugin, OverseerTest, 
DirectSolrConnectionTest, LukeRequestHandlerTest, 
CollectionsAPIDistributedZkTest, LeaderElectionTest, ZkSolrClientTest, 
TestZkChroot, TestRandomDVFaceting, ZkCLITest, ShardRoutingCustomTest, 
TestDistributedSearch, ConfigSetsAPITest, CustomCollectionTest, 
MultiThreadedOCPTest, RemoteQueryErrorTest, SolrXmlInZkTest, 
TestAuthenticationFramework, TestCloudDeleteByQuery, 
TriLevelCompositeIdRoutingTest, HdfsCollectionsAPIDistributedZkTest]
   [junit4] Completed [563/585 (8!)] on J2 in 847.35s, 1 test, 1 error <<< 
FAILURES!

[...truncated 85 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:747: 
The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:684: 
The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:59: 
The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build.xml:233:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/common-build.xml:529:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1457:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1014:
 There were test failures: 585 suites (10 ignored), 2354 tests, 1 suite-level 
error, 3 errors, 5 failures, 31 ignored (16 assumptions) [seed: 
F4FEB0EA8AE0D98C]

Total time: 296 minutes 16 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to