Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.3/8/

7 tests failed.
REGRESSION:  org.apache.solr.TestDistributedSearch.test

Error Message:
Captured an uncaught exception in thread: Thread[id=74293, name=Thread-69129, 
state=RUNNABLE, group=TGRP-TestDistributedSearch]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=74293, name=Thread-69129, state=RUNNABLE, 
group=TGRP-TestDistributedSearch]
        at 
__randomizedtesting.SeedInfo.seed([CE256CC33CAC869:84B669169D36A591]:0)
Caused by: 
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at http://127.0.0.1:36086/dau/g/collection1: 
java.lang.NullPointerException
        at 
org.apache.solr.search.grouping.distributed.responseprocessor.TopGroupsShardResponseProcessor.process(TopGroupsShardResponseProcessor.java:102)
        at 
org.apache.solr.handler.component.QueryComponent.handleGroupedResponses(QueryComponent.java:748)
        at 
org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:731)
        at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:410)
        at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2068)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:669)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:462)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:210)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
        at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:106)
        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
        at 
org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
        at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364)
        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
        at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
        at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
        at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
        at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:499)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
        at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
        at 
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
        at java.lang.Thread.run(Thread.java:745)

        at __randomizedtesting.SeedInfo.seed([CE256CC33CAC869]:0)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:560)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:943)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:958)
        at 
org.apache.solr.TestDistributedSearch$1.run(TestDistributedSearch.java:1110)


REGRESSION:  
org.apache.solr.cloud.AsyncCallRequestStatusResponseTest.testAsyncCallStatusResponse

Error Message:


Stack Trace:
java.lang.AssertionError
        at 
__randomizedtesting.SeedInfo.seed([CE256CC33CAC869:56E333201DF045B0]:0)
        at org.junit.Assert.fail(Assert.java:92)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertTrue(Assert.java:54)
        at 
org.apache.solr.cloud.AsyncCallRequestStatusResponseTest.testAsyncCallStatusResponse(AsyncCallRequestStatusResponseTest.java:38)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:963)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=11397, name=collection0, 
state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=11397, name=collection0, state=RUNNABLE, 
group=TGRP-CollectionsAPIDistributedZkTest]
Caused by: java.lang.RuntimeException: 
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available 
to handle this request:[http://127.0.0.1:58229, http://127.0.0.1:55256, 
http://127.0.0.1:55659, http://127.0.0.1:36930, http://127.0.0.1:49275]
        at __randomizedtesting.SeedInfo.seed([CE256CC33CAC869]:0)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:897)
Caused by: org.apache.solr.client.solrj.SolrServerException: No live 
SolrServers available to handle this request:[http://127.0.0.1:58229, 
http://127.0.0.1:55256, http://127.0.0.1:55659, http://127.0.0.1:36930, 
http://127.0.0.1:49275]
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:355)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1085)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:856)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:799)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:894)
Caused by: 
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at http://127.0.0.1:58229: KeeperErrorCode = Session expired for 
/overseer/collection-queue-work/qn-
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:560)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:376)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:328)
        ... 5 more


REGRESSION:  org.apache.solr.cloud.DistribJoinFromCollectionTest.test

Error Message:
KeeperErrorCode = Session expired for /live_nodes

Stack Trace:
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = 
Session expired for /live_nodes
        at 
__randomizedtesting.SeedInfo.seed([CE256CC33CAC869:84B669169D36A591]:0)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
        at 
org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:328)
        at 
org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:325)
        at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
        at 
org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:325)
        at 
org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:561)
        at 
org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:239)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.ensureAllReplicasAreActive(AbstractFullDistribZkTestBase.java:1807)
        at 
org.apache.solr.cloud.DistribJoinFromCollectionTest.test(DistribJoinFromCollectionTest.java:95)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:963)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
        at java.lang.Thread.run(Thread.java:745)


REGRESSION:  org.apache.solr.cloud.hdfs.HdfsBasicDistributedZkTest.test

Error Message:
Error from server at http://127.0.0.1:57397/vo_v/collection1: ClusterState says 
we are the leader (http://127.0.0.1:57397/vo_v/collection1), but locally we 
don't think so. Request came from null

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at http://127.0.0.1:57397/vo_v/collection1: ClusterState says we 
are the leader (http://127.0.0.1:57397/vo_v/collection1), but locally we don't 
think so. Request came from null
        at 
__randomizedtesting.SeedInfo.seed([CE256CC33CAC869:84B669169D36A591]:0)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:560)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:152)
        at 
org.apache.solr.BaseDistributedSearchTestCase.add(BaseDistributedSearchTestCase.java:511)
        at 
org.apache.solr.cloud.BasicDistributedZkTest.testUpdateProcessorsRunOnlyOnce(BasicDistributedZkTest.java:639)
        at 
org.apache.solr.cloud.BasicDistributedZkTest.test(BasicDistributedZkTest.java:350)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:963)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
        at java.lang.Thread.run(Thread.java:745)


REGRESSION:  org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest.test

Error Message:
shard3 is not consistent.  Got 247 from 
https://127.0.0.1:56737/_zim/hi/collection1lastClient and got 164 from 
https://127.0.0.1:53641/_zim/hi/collection1

Stack Trace:
java.lang.AssertionError: shard3 is not consistent.  Got 247 from 
https://127.0.0.1:56737/_zim/hi/collection1lastClient and got 164 from 
https://127.0.0.1:53641/_zim/hi/collection1
        at 
__randomizedtesting.SeedInfo.seed([CE256CC33CAC869:84B669169D36A591]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1244)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1223)
        at 
org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test(ChaosMonkeySafeLeaderTest.java:165)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:963)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=79096, name=collection2, 
state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=79096, name=collection2, state=RUNNABLE, 
group=TGRP-HdfsCollectionsAPIDistributedZkTest]
Caused by: 
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at http://127.0.0.1:48997: Could not find collection : 
awholynewstresscollection_collection2_3
        at __randomizedtesting.SeedInfo.seed([CE256CC33CAC869]:0)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:560)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:376)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:328)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1085)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:856)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:799)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:894)




Build Log:
[...truncated 10509 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest
   [junit4]   2> Creating dataDir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_CE256CC33CAC869-001/init-core-data-001
   [junit4]   2> 556635 INFO  
(SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[CE256CC33CAC869]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 556635 INFO  
(SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[CE256CC33CAC869]-worker) [    ] 
o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: 
/_zim/hi
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 556674 WARN  
(SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[CE256CC33CAC869]-worker) [    ] 
o.a.h.m.i.MetricsConfig Cannot locate configuration: tried 
hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 556683 WARN  
(SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[CE256CC33CAC869]-worker) [    ] 
o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 556688 INFO  
(SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[CE256CC33CAC869]-worker) [    ] 
o.m.log jetty-6.1.26
   [junit4]   2> 556704 INFO  
(SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[CE256CC33CAC869]-worker) [    ] 
o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/hdfs
 to ./temp/Jetty_localhost_52557_hdfs____p1gy2s/webapp
   [junit4]   2> 556829 INFO  
(SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[CE256CC33CAC869]-worker) [    ] 
o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 557314 INFO  
(SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[CE256CC33CAC869]-worker) [    ] 
o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:52557
   [junit4]   2> 557476 WARN  
(SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[CE256CC33CAC869]-worker) [    ] 
o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 557477 INFO  
(SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[CE256CC33CAC869]-worker) [    ] 
o.m.log jetty-6.1.26
   [junit4]   2> 557493 INFO  
(SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[CE256CC33CAC869]-worker) [    ] 
o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode
 to ./temp/Jetty_localhost_53677_datanode____.3qnrvl/webapp
   [junit4]   2> 557691 INFO  
(SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[CE256CC33CAC869]-worker) [    ] 
o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 558153 INFO  
(SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[CE256CC33CAC869]-worker) [    ] 
o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:53677
   [junit4]   2> 558187 WARN  
(SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[CE256CC33CAC869]-worker) [    ] 
o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 558189 INFO  
(SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[CE256CC33CAC869]-worker) [    ] 
o.m.log jetty-6.1.26
   [junit4]   2> 558205 INFO  
(SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[CE256CC33CAC869]-worker) [    ] 
o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode
 to ./temp/Jetty_localhost_51691_datanode____pjddp5/webapp
   [junit4]   2> 558252 INFO  (IPC Server handler 3 on 52095) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-06bd408f-01e8-4260-a36a-1ab09003185d node DatanodeRegistration(127.0.0.1, 
datanodeUuid=8e1655f6-c766-4907-b501-b11a31068378, infoPort=53677, 
ipcPort=44740, storageInfo=lv=-56;cid=testClusterID;nsid=2032221109;c=0), 
blocks: 0, hasStaleStorages: true, processing time: 1 msecs
   [junit4]   2> 558253 INFO  (IPC Server handler 3 on 52095) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-3b9c02a3-3c16-44ce-9284-15b17e4693ac node DatanodeRegistration(127.0.0.1, 
datanodeUuid=8e1655f6-c766-4907-b501-b11a31068378, infoPort=53677, 
ipcPort=44740, storageInfo=lv=-56;cid=testClusterID;nsid=2032221109;c=0), 
blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 558349 INFO  
(SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[CE256CC33CAC869]-worker) [    ] 
o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 558860 INFO  
(SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[CE256CC33CAC869]-worker) [    ] 
o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:51691
   [junit4]   2> 558949 INFO  (IPC Server handler 4 on 52095) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-be9f15fb-0c1a-41e5-ae5d-2ce9c8e8e5dd node DatanodeRegistration(127.0.0.1, 
datanodeUuid=8e5ecb1a-4c78-4b3d-9158-1f33fa781b8d, infoPort=51691, 
ipcPort=55865, storageInfo=lv=-56;cid=testClusterID;nsid=2032221109;c=0), 
blocks: 0, hasStaleStorages: true, processing time: 0 msecs
   [junit4]   2> 558950 INFO  (IPC Server handler 4 on 52095) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-108c8116-2f3d-4ce1-9765-c5e8d338dd30 node DatanodeRegistration(127.0.0.1, 
datanodeUuid=8e5ecb1a-4c78-4b3d-9158-1f33fa781b8d, infoPort=51691, 
ipcPort=55865, storageInfo=lv=-56;cid=testClusterID;nsid=2032221109;c=0), 
blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 559025 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 559025 INFO  (Thread-15766) [    ] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 559025 INFO  (Thread-15766) [    ] o.a.s.c.ZkTestServer 
Starting server
   [junit4]   2> 559125 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.ZkTestServer start zk server on port:51301
   [junit4]   2> 559127 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 559128 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 559132 INFO  (zkCallback-193-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@703bd33f 
name:ZooKeeperConnection Watcher:127.0.0.1:51301 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 559132 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 559132 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 559132 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 559137 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 559138 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 559139 INFO  (zkCallback-194-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@27a0d7f6 
name:ZooKeeperConnection Watcher:127.0.0.1:51301/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 559139 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 559139 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 559139 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 559144 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 559146 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 559149 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 559151 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
 to /configs/conf1/solrconfig.xml
   [junit4]   2> 559151 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 559154 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/schema15.xml
 to /configs/conf1/schema.xml
   [junit4]   2> 559155 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 559157 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
 to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 559157 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.SolrZkClient makePath: 
/configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 559159 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
 to /configs/conf1/stopwords.txt
   [junit4]   2> 559159 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 559161 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/protwords.txt
 to /configs/conf1/protwords.txt
   [junit4]   2> 559161 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 559163 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/currency.xml
 to /configs/conf1/currency.xml
   [junit4]   2> 559163 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 559166 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
 to /configs/conf1/enumsConfig.xml
   [junit4]   2> 559166 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 559170 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
 to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 559170 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 559173 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
 to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 559173 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 559174 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
 to /configs/conf1/old_synonyms.txt
   [junit4]   2> 559175 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 559179 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
 to /configs/conf1/synonyms.txt
   [junit4]   2> 559179 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 559183 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 559184 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 559185 INFO  (zkCallback-195-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@5b761d5a 
name:ZooKeeperConnection Watcher:127.0.0.1:51301/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 559185 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 559186 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 559485 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.SolrTestCaseJ4 Writing core.properties file to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_CE256CC33CAC869-001/control-001/cores/collection1
   [junit4]   2> 559487 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.e.j.s.Server jetty-9.2.11.v20150529
   [junit4]   2> 559489 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@7ddcc13e{/_zim/hi,null,AVAILABLE}
   [junit4]   2> 559494 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.e.j.s.ServerConnector Started 
ServerConnector@7afc368d{SSL-http/1.1}{127.0.0.1:46910}
   [junit4]   2> 559494 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.e.j.s.Server Started @562692ms
   [junit4]   2> 559495 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/_zim/hi, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_CE256CC33CAC869-001/control-001/cores,
 
solr.data.dir=hdfs://localhost:52095/hdfs__localhost_52095__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-5.3_solr_build_solr-core_test_J0_temp_solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_CE256CC33CAC869-001_tempDir-002_control_data,
 hostPort=46910}
   [junit4]   2> 559495 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): 
sun.misc.Launcher$AppClassLoader@1fe91485
   [junit4]   2> 559495 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_CE256CC33CAC869-001/control-001/'
   [junit4]   2> 559512 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 559513 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 559516 INFO  (zkCallback-196-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@453510db 
name:ZooKeeperConnection Watcher:127.0.0.1:51301/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 559516 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 559516 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 559519 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 559520 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_CE256CC33CAC869-001/control-001/solr.xml
   [junit4]   2> 559536 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.CoresLocator Config-defined core root directory: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_CE256CC33CAC869-001/control-001/cores
   [junit4]   2> 559536 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.CoreContainer New CoreContainer 224294256
   [junit4]   2> 559536 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.CoreContainer Loading cores into CoreContainer 
[instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_CE256CC33CAC869-001/control-001/]
   [junit4]   2> 559536 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.CoreContainer loading shared library: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_CE256CC33CAC869-001/control-001/lib
   [junit4]   2> 559536 WARN  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.SolrResourceLoader Can't find (or read) directory to add to 
classloader: lib (resolved as: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_CE256CC33CAC869-001/control-001/lib).
   [junit4]   2> 559545 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme 
: ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 
10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 
5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 559548 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 559550 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 559550 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.l.LogWatcher Registering Log Listener [Log4j 
(org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 559550 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51301/solr
   [junit4]   2> 559550 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 559550 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 559552 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 559553 INFO  (zkCallback-198-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@5a8b238a 
name:ZooKeeperConnection Watcher:127.0.0.1:51301 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 559554 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 559554 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 559558 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.ConnectionManager Waiting for 
client to connect to ZooKeeper
   [junit4]   2> 559559 INFO  
(zkCallback-199-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@613e9592 
name:ZooKeeperConnection Watcher:127.0.0.1:51301/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 559559 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.ConnectionManager Client is 
connected to ZooKeeper
   [junit4]   2> 559560 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/queue
   [junit4]   2> 559563 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-queue-work
   [junit4]   2> 559566 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-map-running
   [junit4]   2> 559568 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-map-completed
   [junit4]   2> 559570 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-map-failure
   [junit4]   2> 559572 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 559574 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 559576 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.SolrZkClient makePath: 
/clusterstate.json
   [junit4]   2> 559577 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.SolrZkClient makePath: 
/security.json
   [junit4]   2> 559578 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:46910__zim%2Fhi
   [junit4]   2> 559578 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.SolrZkClient makePath: 
/live_nodes/127.0.0.1:46910__zim%2Fhi
   [junit4]   2> 559580 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer_elect
   [junit4]   2> 559581 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer_elect/election
   [junit4]   2> 559583 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 559583 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.LeaderElector Joined leadership 
election with path: 
/overseer_elect/election/94364008793571333-127.0.0.1:46910__zim%2Fhi-n_0000000000
   [junit4]   2> 559584 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.OverseerElectionContext I am going to 
be the leader 127.0.0.1:46910__zim%2Fhi
   [junit4]   2> 559584 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer_elect/leader
   [junit4]   2> 559585 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.Overseer Overseer 
(id=94364008793571333-127.0.0.1:46910__zim%2Fhi-n_0000000000) starting
   [junit4]   2> 559587 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/queue-work
   [junit4]   2> 559593 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.OverseerAutoReplicaFailoverThread 
Starting OverseerAutoReplicaFailoverThread 
autoReplicaFailoverWorkLoopDelay=10000 
autoReplicaFailoverWaitAfterExpiration=30000 
autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 559593 INFO  
(OverseerCollectionProcessor-94364008793571333-127.0.0.1:46910__zim%2Fhi-n_0000000000)
 [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.OverseerCollectionProcessor Process 
current queue of collection creations
   [junit4]   2> 559594 INFO  
(OverseerStateUpdate-94364008793571333-127.0.0.1:46910__zim%2Fhi-n_0000000000) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.Overseer Starting to work on the main 
queue
   [junit4]   2> 559594 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.ZkStateReader Updating cluster 
state from ZooKeeper... 
   [junit4]   2> 559615 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.CoreContainer Security conf doesn't 
exist. Skipping setup for authorization module.
   [junit4]   2> 559615 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.CoreContainer No authentication 
plugin used.
   [junit4]   2> 559616 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.CoresLocator Looking for core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_CE256CC33CAC869-001/control-001/cores
   [junit4]   2> 559617 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.SolrCore Created CoreDescriptor: 
{loadOnStartup=true, collection=control_collection, config=solrconfig.xml, 
instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_CE256CC33CAC869-001/control-001/cores/collection1,
 schema=schema.xml, 
absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_CE256CC33CAC869-001/control-001/cores/collection1/,
 shard=, coreNodeName=, name=collection1, transient=false, dataDir=data/, 
configSetProperties=configsetprops.json}
   [junit4]   2> 559617 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.CoresLocator Found core collection1 
in 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_CE256CC33CAC869-001/control-001/cores/collection1/
   [junit4]   2> 559617 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.CoresLocator Found 1 core definitions
   [junit4]   2> 559619 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.s.SolrDispatchFilter 
user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0
   [junit4]   2> 559619 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.s.SolrDispatchFilter 
SolrDispatchFilter.init() done
   [junit4]   2> 559621 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] 
o.a.s.c.ZkController publishing state=down
   [junit4]   2> 559621 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] 
o.a.s.c.ZkController numShards not found on descriptor - reading it from system 
property
   [junit4]   2> 559622 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] 
o.a.s.c.ZkController look for our core node name
   [junit4]   2> 559622 INFO  
(zkCallback-199-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.DistributedQueue NodeChildrenChanged 
fired on path /overseer/queue state SyncConnected
   [junit4]   2> 559623 INFO  
(OverseerStateUpdate-94364008793571333-127.0.0.1:46910__zim%2Fhi-n_0000000000) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.Overseer processMessage: queueSize: 
1, message = {
   [junit4]   2>   "base_url":"https://127.0.0.1:46910/_zim/hi";,
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "state":"down",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "node_name":"127.0.0.1:46910__zim%2Fhi",
   [junit4]   2>   "roles":null} current state version: 0
   [junit4]   2> 559624 INFO  
(OverseerStateUpdate-94364008793571333-127.0.0.1:46910__zim%2Fhi-n_0000000000) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.o.ReplicaMutator Update state 
numShards=1 message={
   [junit4]   2>   "base_url":"https://127.0.0.1:46910/_zim/hi";,
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "state":"down",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "node_name":"127.0.0.1:46910__zim%2Fhi",
   [junit4]   2>   "roles":null}
   [junit4]   2> 559624 INFO  
(OverseerStateUpdate-94364008793571333-127.0.0.1:46910__zim%2Fhi-n_0000000000) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.o.ClusterStateMutator building a new 
cName: control_collection
   [junit4]   2> 559624 INFO  
(OverseerStateUpdate-94364008793571333-127.0.0.1:46910__zim%2Fhi-n_0000000000) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.o.ReplicaMutator Assigning new node 
to shard shard=shard1
   [junit4]   2> 559625 INFO  
(zkCallback-199-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.ZkStateReader A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 559628 INFO  
(zkCallback-199-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.ZkStateReader Updated cluster state 
version to 1
   [junit4]   2> 560622 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] 
o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 560622 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] 
o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 560623 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] 
o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 560623 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] 
o.a.s.c.c.ZkStateReader Load collection config 
from:/collections/control_collection
   [junit4]   2> 560624 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] 
o.a.s.c.c.ZkStateReader path=/collections/control_collection configName=conf1 
specified config exists in ZooKeeper
   [junit4]   2> 560624 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] 
o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_CE256CC33CAC869-001/control-001/cores/collection1/'
   [junit4]   2> 560642 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] 
o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 560648 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] 
o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 560661 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 5.3.0
   [junit4]   2> 560697 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] 
o.a.s.c.Config Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 560698 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] 
o.a.s.s.IndexSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 560705 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] 
o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 560933 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] 
o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 560934 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] 
o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 560936 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] 
o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 560939 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] 
o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 560964 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] 
o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default 
properties: Can't find resource 'configsetprops.json' in classpath or 
'/configs/conf1', 
cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0
   [junit4]   2> 560965 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] 
o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from 
collection control_collection
   [junit4]   2> 560965 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore org.apache.solr.core.HdfsDirectoryFactory
   [junit4]   2> 560965 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.HdfsDirectoryFactory 
solr.hdfs.home=hdfs://localhost:52095/solr_hdfs_home
   [junit4]   2> 560965 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication 
disabled
   [junit4]   2> 560965 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_CE256CC33CAC869-001/control-001/cores/collection1/],
 dataDir=[null]
   [junit4]   2> 560966 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr 
mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@23a70112
   [junit4]   2> 560966 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:52095/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 560984 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.s.h.HdfsLocalityReporter Registering direcotry 
hdfs://localhost:52095/solr_hdfs_home/control_collection/core_node1/data for 
locality metrics.
   [junit4]   2> 560984 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for 
hdfs://localhost:52095/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 560985 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore New index directory detected: old=null 
new=hdfs://localhost:52095/solr_hdfs_home/control_collection/core_node1/data/index/
   [junit4]   2> 560996 WARN  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory 
'hdfs:/localhost:52095/solr_hdfs_home/control_collection/core_node1/data/index' 
doesn't exist. Creating new index...
   [junit4]   2> 560996 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:52095/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 561005 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] 
with direct memory allocation set to [true]
   [junit4]   2> 561005 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, 
slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 561005 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS 
BlockCache
   [junit4]   2> 561013 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 561013 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.s.h.HdfsLocalityReporter Registering direcotry 
hdfs://localhost:52095/solr_hdfs_home/control_collection/core_node1/data/index 
for locality metrics.
   [junit4]   2> 561013 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for 
hdfs://localhost:52095/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 561013 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.MockRandomMergePolicy: 
org.apache.lucene.index.MockRandomMergePolicy@5d0041cd
   [junit4]   2> 561050 INFO  (IPC Server handler 8 on 52095) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54084 is 
added to blk_1073741825_1001{blockUCState=UNDER_CONSTRUCTION, 
primaryNodeIndex=-1, 
replicas=[ReplicaUnderConstruction[[DISK]DS-06bd408f-01e8-4260-a36a-1ab09003185d:NORMAL:127.0.0.1:37256|RBW],
 
ReplicaUnderConstruction[[DISK]DS-108c8116-2f3d-4ce1-9765-c5e8d338dd30:NORMAL:127.0.0.1:54084|FINALIZED]]}
 size 0
   [junit4]   2> 561052 INFO  (IPC Server handler 9 on 52095) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37256 is 
added to blk_1073741825_1001{blockUCState=UNDER_CONSTRUCTION, 
primaryNodeIndex=-1, 
replicas=[ReplicaUnderConstruction[[DISK]DS-108c8116-2f3d-4ce1-9765-c5e8d338dd30:NORMAL:127.0.0.1:54084|FINALIZED],
 
ReplicaUnderConstruction[[DISK]DS-3b9c02a3-3c16-44ce-9284-15b17e4693ac:NORMAL:127.0.0.1:37256|FINALIZED]]}
 size 0
   [junit4]   2> 561055 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2>        
commit{dir=NRTCachingDirectory(BlockDirectory(HdfsDirectory@b8369659 
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@381847d9); 
maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 561055 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore newest commit generation = 1
   [junit4]   2> 561057 INFO  
(OldIndexDirectoryCleanupThreadForCore-collection1) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup 
for core collection1 in 
hdfs://localhost:52095/solr_hdfs_home/control_collection/core_node1/data/
   [junit4]   2> 561061 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating 
updateRequestProcessorChain "nodistrib"
   [junit4]   2> 561061 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating 
updateRequestProcessorChain "dedupe"
   [junit4]   2> 561063 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting 
DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 561063 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating 
updateRequestProcessorChain "stored_sig"
   [junit4]   2> 561063 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting 
DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 561063 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating 
updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 561064 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating 
updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 561067 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting 
DistributedUpdateProcessorFactory into updateRequestProcessorChain 
"distrib-dup-test-chain-implicit"
   [junit4]   2> 561068 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as 
default, creating implicit default
   [junit4]   2> 561073 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 561074 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 561076 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 561077 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 561081 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.RequestHandlers Registered paths: 
/update/json/docs,standard,/admin/logging,/replication,/schema,/config,/admin/luke,/admin/ping,/update/csv,/update/json,/update,/admin/plugins,/admin/properties,/get,/admin/segments,/admin/system,/admin/mbeans,/admin/threads,/admin/file
   [junit4]   2> 561082 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore Using default statsCache cache: 
org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 561083 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: 
org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 561083 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.UpdateLog Initializing HdfsUpdateLog: dataDir= 
defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 
tlogDfsReplication=2
   [junit4]   2> 561095 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore Hard AutoCommit: disabled
   [junit4]   2> 561095 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore Soft AutoCommit: disabled
   [junit4]   2> 561097 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: 
minMergeSize=1677721, mergeFactor=23, maxMergeSize=2147483648, 
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, 
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=1.0]
   [junit4]   2> 561105 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2>        
commit{dir=NRTCachingDirectory(BlockDirectory(HdfsDirectory@b8369659 
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@381847d9); 
maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 561105 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore newest commit generation = 1
   [junit4]   2> 561107 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@1a2fb6b6[collection1] 
main
   [junit4]   2> 561107 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.c.ZkStateReader Load collection config 
from:/collections/control_collection
   [junit4]   2> 561109 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.c.ZkStateReader path=/collections/control_collection 
configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 561109 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based 
storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 561110 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO 
with znodeBase: /configs/conf1
   [junit4]   2> 561110 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 561110 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 561111 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode 
/configs/conf1/_rest_managed.json
   [junit4]   2> 561111 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path 
_rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 561111 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 561111 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 561113 INFO  
(searcherExecutor-1324-thread-1-processing-x:collection1 s:shard1 r:core_node1 
c:control_collection n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore 
[collection1] Registered new searcher Searcher@1a2fb6b6[collection1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 561115 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed 
version buckets
   [junit4]   2> 561115 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 
256 version buckets from index
   [junit4]   2> 561115 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed 
version bucket highest value from index
   [junit4]   2> 561115 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent 
updates, using new clock 1509824142886567936
   [junit4]   2> 561115 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.UpdateLog Took 0 ms to seed version buckets with highest 
version 1509824142886567936
   [junit4]   2> 561115 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 561116 INFO  
(coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) 
[n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 561120 INFO  
(coreZkRegister-1317-thread-1-processing-x:collection1 s:shard1 r:core_node1 
c:control_collection n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController 
Register replica - core:collection1 address:https://127.0.0.1:46910/_zim/hi 
collection:control_collection shard:shard1
   [junit4]   2> 561123 INFO  
(coreZkRegister-1317-thread-1-processing-x:collection1 s:shard1 r:core_node1 
c:control_collection n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.c.SolrZkClient makePath: 
/collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 561125 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 561125 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 561128 INFO  (zkCallback-201-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@46f66847 
name:ZooKeeperConnection Watcher:127.0.0.1:51301/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 561128 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 561128 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 561128 INFO  
(TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] 
o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 561134 INFO  
(coreZkRegister-1317-thread-1-processing-x:collection1 s:shard1 r:core_node1 
c:control_collection n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector 
Joined leadership election with path: 
/collections/control_collection/leader_elect/shard1/election/94364008793571333-core_node1-n_0000000000
   [junit4]   2> 561135 INFO  
(coreZkRegister-1317-thread-1-processing-x:collection1 s:shard1 r:core_node1 
c:control_collection n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.ShardLeaderElectionContext Running the leader process for shard shard1
   [junit4]   2> 561137 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-

[...truncated too long message...]

06)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.lib.MethodMetric$2.snapshot(MethodMetric.java:111)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.lib.MethodMetric.snapshot(MethodMetric.java:144)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.lib.MetricsRegistry.snapshot(MetricsRegistry.java:387)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.lib.MetricsSourceBuilder$1.getMetrics(MetricsSourceBuilder.java:79)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(MetricsSourceAdapter.java:195)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateJmxCache(MetricsSourceAdapter.java:172)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMBeanInfo(MetricsSourceAdapter.java:151)
   [junit4]   2>        at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getClassName(DefaultMBeanServerInterceptor.java:1804)
   [junit4]   2>        at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.safeGetClassName(DefaultMBeanServerInterceptor.java:1595)
   [junit4]   2>        at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.checkMBeanPermission(DefaultMBeanServerInterceptor.java:1813)
   [junit4]   2>        at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:430)
   [junit4]   2>        at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
   [junit4]   2>        at 
com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:81)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stopMBeans(MetricsSourceAdapter.java:227)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stop(MetricsSourceAdapter.java:212)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stopSources(MetricsSystemImpl.java:461)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:212)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.shutdown(MetricsSystemImpl.java:592)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdownInstance(DefaultMetricsSystem.java:72)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdown(DefaultMetricsSystem.java:68)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics.shutdown(NameNodeMetrics.java:145)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:822)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1720)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1699)
   [junit4]   2>        at 
org.apache.solr.cloud.hdfs.HdfsTestUtil.teardownClass(HdfsTestUtil.java:197)
   [junit4]   2>        at 
org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.teardownClass(HdfsCollectionsAPIDistributedZkTest.java:50)
   [junit4]   2>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
   [junit4]   2>        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   2>        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>        at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:799)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2>        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2>        at 
org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:198)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3291)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:6223)
   [junit4]   2>        ... 53 more
   [junit4]   2> 2939348 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[CE256CC33CAC869]-worker) 
[n:127.0.0.1:45483_ c:addReplicaColl s:shard2 r:core_node6 
x:addReplicaColl_shard2_replica3] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> Aug 18, 2015 3:28:06 AM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 2 leaked 
thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene53), 
sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=es_CR, 
timezone=Etc/GMT+2
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 
1.7.0_72 (64-bit)/cpus=4,threads=2,free=146510560,total=533200896
   [junit4]   2> NOTE: All tests run in this JVM: 
[TestSchemaSimilarityResource, TestBinaryField, TestRealTimeGet, TestJoin, 
BasicZkTest, StressHdfsTest, TestStressLucene, LeaderElectionIntegrationTest, 
TestSolr4Spatial, TestReversedWildcardFilterFactory, TestRebalanceLeaders, 
TestSolrDeletionPolicy1, FastVectorHighlighterTest, HdfsDirectoryTest, 
IgnoreCommitOptimizeUpdateProcessorFactoryTest, MultiTermTest, 
ConcurrentDeleteAndCreateCollectionTest, LeaderInitiatedRecoveryOnCommitTest, 
TestConfigOverlay, ClusterStateTest, BitVectorTest, TestDocumentBuilder, 
URLClassifyProcessorTest, TestWordDelimiterFilterFactory, TestPivotHelperCode, 
ExitableDirectoryReaderTest, HdfsChaosMonkeySafeLeaderTest, 
UniqFieldsUpdateProcessorFactoryTest, SuggestComponentTest, 
DistribJoinFromCollectionTest, ChaosMonkeyNothingIsSafeTest, 
DefaultValueUpdateProcessorTest, SolrRequestParserTest, TestReqParamsAPI, 
CloudMLTQParserTest, CoreAdminRequestStatusTest, 
DistributedSpellCheckComponentTest, DistributedFacetPivotSmallAdvancedTest, 
TestCloudManagedSchemaConcurrent, TestDistributedSearch, 
BasicDistributedZkTest, TestChildDocTransformer, 
TestManagedSynonymFilterFactory, TestNamedUpdateProcessors, SuggesterFSTTest, 
PeerSyncTest, DistanceUnitsTest, CurrencyFieldXmlFileTest, 
VMParamsZkACLAndCredentialsProvidersTest, TestCursorMarkWithoutUniqueKey, 
JavabinLoaderTest, AddSchemaFieldsUpdateProcessorFactoryTest, 
CoreMergeIndexesAdminHandlerTest, TermsComponentTest, 
BigEndianAscendingWordDeserializerTest, MBeansHandlerTest, 
CSVRequestHandlerTest, TestRemoteStreaming, TestBadConfig, TestCollectionAPI, 
TestArbitraryIndexDir, DeleteInactiveReplicaTest, TestRandomFaceting, 
HdfsDirectoryFactoryTest, TestBlendedInfixSuggestions, SuggesterWFSTTest, 
TestPhraseSuggestions, HdfsCollectionsAPIDistributedZkTest]
   [junit4] Completed [380/531] on J0 in 440.43s, 1 test, 1 error <<< FAILURES!

[...truncated 515 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/build.xml:733: 
The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/build.xml:670: 
The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/build.xml:59: 
The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build.xml:230:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/common-build.xml:524:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/lucene/common-build.xml:1449:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/lucene/common-build.xml:1003:
 There were test failures: 531 suites, 2104 tests, 5 errors, 2 failures, 26 
ignored (14 assumptions)

Total time: 189 minutes 56 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to