Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1471/

8 tests failed.
FAILED:  
junit.framework.TestSuite.org.apache.solr.ltr.feature.TestFieldLengthFeature

Error Message:
1 thread leaked from SUITE scope at 
org.apache.solr.ltr.feature.TestFieldLengthFeature:     1) Thread[id=88, 
name=qtp1192070498-88, state=TIMED_WAITING, group=TGRP-TestFieldLengthFeature]  
       at sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) 
        at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE 
scope at org.apache.solr.ltr.feature.TestFieldLengthFeature: 
   1) Thread[id=88, name=qtp1192070498-88, state=TIMED_WAITING, 
group=TGRP-TestFieldLengthFeature]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
        at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
        at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([A539D143B4BF2303]:0)


FAILED:  
junit.framework.TestSuite.org.apache.solr.ltr.feature.TestFieldLengthFeature

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=88, 
name=qtp1192070498-88, state=TIMED_WAITING, group=TGRP-TestFieldLengthFeature]  
       at sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) 
        at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie 
threads that couldn't be terminated:
   1) Thread[id=88, name=qtp1192070498-88, state=TIMED_WAITING, 
group=TGRP-TestFieldLengthFeature]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
        at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
        at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([A539D143B4BF2303]:0)


FAILED:  org.apache.solr.cloud.MoveReplicaHDFSTest.testFailedMove

Error Message:
No live SolrServers available to handle this 
request:[https://127.0.0.1:38320/solr/MoveReplicaHDFSTest_failed_coll_true, 
https://127.0.0.1:43978/solr/MoveReplicaHDFSTest_failed_coll_true]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available 
to handle this 
request:[https://127.0.0.1:38320/solr/MoveReplicaHDFSTest_failed_coll_true, 
https://127.0.0.1:43978/solr/MoveReplicaHDFSTest_failed_coll_true]
        at 
__randomizedtesting.SeedInfo.seed([970C435508514221:3DC190A7BF8297F1]:0)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:462)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1104)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:991)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
        at 
org.apache.solr.cloud.MoveReplicaTest.testFailedMove(MoveReplicaTest.java:309)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: 
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at 
https://127.0.0.1:38320/solr/MoveReplicaHDFSTest_failed_coll_true: no servers 
hosting shard: shard1
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:436)
        ... 46 more


FAILED:  org.apache.solr.cloud.RollingRestartTest.test

Error Message:
Unable to restart (#5): CloudJettyRunner 
[url=https://127.0.0.1:45142/eo/r/collection1_shard2_replica_n45]

Stack Trace:
java.lang.AssertionError: Unable to restart (#5): CloudJettyRunner 
[url=https://127.0.0.1:45142/eo/r/collection1_shard2_replica_n45]
        at 
__randomizedtesting.SeedInfo.seed([970C435508514221:1F587C8FA6AD2FD9]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at 
org.apache.solr.cloud.RollingRestartTest.restartWithRolesTest(RollingRestartTest.java:103)
        at 
org.apache.solr.cloud.RollingRestartTest.test(RollingRestartTest.java:52)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testListeners

Error Message:


Stack Trace:
java.lang.AssertionError
        at 
__randomizedtesting.SeedInfo.seed([970C435508514221:7ACCD445B53DDFF8]:0)
        at org.junit.Assert.fail(Assert.java:92)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertTrue(Assert.java:54)
        at 
org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testListeners(TriggerIntegrationTest.java:1053)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsRecoveryZkTest

Error Message:
ObjectTracker found 1 object(s) that were not released!!! [HdfsTransactionLog] 
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: 
org.apache.solr.update.HdfsTransactionLog  at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
  at 
org.apache.solr.update.HdfsTransactionLog.<init>(HdfsTransactionLog.java:132)  
at org.apache.solr.update.HdfsUpdateLog.init(HdfsUpdateLog.java:203)  at 
org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:161)  at 
org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:116)  at 
org.apache.solr.update.DirectUpdateHandler2.<init>(DirectUpdateHandler2.java:113)
  at sun.reflect.GeneratedConstructorAccessor445.newInstance(Unknown Source)  
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)  at 
org.apache.solr.core.SolrCore.createInstance(SolrCore.java:793)  at 
org.apache.solr.core.SolrCore.createUpdateHandler(SolrCore.java:855)  at 
org.apache.solr.core.SolrCore.initUpdateHandler(SolrCore.java:1108)  at 
org.apache.solr.core.SolrCore.<init>(SolrCore.java:978)  at 
org.apache.solr.core.SolrCore.<init>(SolrCore.java:863)  at 
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1043)
  at org.apache.solr.core.CoreContainer.lambda$load$13(CoreContainer.java:644)  
at 
com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
 at java.lang.Thread.run(Thread.java:748)  

Stack Trace:
java.lang.AssertionError: ObjectTracker found 1 object(s) that were not 
released!!! [HdfsTransactionLog]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: 
org.apache.solr.update.HdfsTransactionLog
        at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at 
org.apache.solr.update.HdfsTransactionLog.<init>(HdfsTransactionLog.java:132)
        at org.apache.solr.update.HdfsUpdateLog.init(HdfsUpdateLog.java:203)
        at org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:161)
        at org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:116)
        at 
org.apache.solr.update.DirectUpdateHandler2.<init>(DirectUpdateHandler2.java:113)
        at sun.reflect.GeneratedConstructorAccessor445.newInstance(Unknown 
Source)
        at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:793)
        at org.apache.solr.core.SolrCore.createUpdateHandler(SolrCore.java:855)
        at org.apache.solr.core.SolrCore.initUpdateHandler(SolrCore.java:1108)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:978)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:863)
        at 
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1043)
        at 
org.apache.solr.core.CoreContainer.lambda$load$13(CoreContainer.java:644)
        at 
com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)


        at __randomizedtesting.SeedInfo.seed([970C435508514221]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNull(Assert.java:551)
        at 
org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:295)
        at sun.reflect.GeneratedMethodAccessor123.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:897)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest.test

Error Message:
There are still nodes recoverying - waited for 600 seconds

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 600 
seconds
        at 
__randomizedtesting.SeedInfo.seed([970C435508514221:1F587C8FA6AD2FD9]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:185)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:921)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1478)
        at 
org.apache.solr.cloud.TlogReplayBufferedWhileIndexingTest.test(TlogReplayBufferedWhileIndexingTest.java:99)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.search.mlt.SimpleMLTQParserTest.doTest

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
        at 
__randomizedtesting.SeedInfo.seed([970C435508514221:3048FBF165EA5198]:0)
        at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:902)
        at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:862)
        at 
org.apache.solr.search.mlt.SimpleMLTQParserTest.doTest(SimpleMLTQParserTest.java:82)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: REQUEST FAILED: 
xpath=//result/doc[1]/str[@name='id'][.='13']
        xml response was: <?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int 
name="QTime">19</int></lst><result name="response" numFound="16" 
start="0"><doc><str name="id">14</str><arr 
name="range_facet_l"><long>14</long></arr><int name="id_i1">14</int><arr 
name="range_facet_l_dv"><long>14</long></arr><arr 
name="range_facet_i_dv"><int>14</int></arr><arr name="lowerfilt"><str>The quote 
red fox jumped over the lazy brown dogs.</str></arr><arr 
name="lowerfilt1and2"><str>The quote red fox jumped over the lazy brown 
dogs.</str></arr><arr name="multiDefault"><str>muLti-Default</str></arr><int 
name="intDefault">42</int><date 
name="timestamp">2018-02-06T12:58:12.157Z</date><int 
name="intDvoDefault">42</int></doc><doc><str name="id">15</str><arr 
name="range_facet_l"><long>15</long></arr><int name="id_i1">15</int><arr 
name="range_facet_l_dv"><long>15</long></arr><arr 
name="range_facet_i_dv"><int>15</int></arr><arr name="lowerfilt"><str>The fat 
red fox jumped over the lazy brown dogs.</str></arr><arr 
name="lowerfilt1and2"><str>The fat red fox jumped over the lazy brown 
dogs.</str></arr><arr name="multiDefault"><str>muLti-Default</str></arr><int 
name="intDefault">42</int><date 
name="timestamp">2018-02-06T12:58:12.228Z</date><int 
name="intDvoDefault">42</int></doc><doc><str name="id">16</str><arr 
name="range_facet_l"><long>16</long></arr><int name="id_i1">16</int><arr 
name="range_facet_l_dv"><long>16</long></arr><arr 
name="range_facet_i_dv"><int>16</int></arr><arr name="lowerfilt"><str>The slim 
red fox jumped over the lazy brown dogs.</str></arr><arr 
name="lowerfilt1and2"><str>The slim red fox jumped over the lazy brown 
dogs.</str></arr><arr name="multiDefault"><str>muLti-Default</str></arr><int 
name="intDefault">42</int><date 
name="timestamp">2018-02-06T12:58:12.228Z</date><int 
name="intDvoDefault">42</int></doc><doc><str name="id">18</str><arr 
name="range_facet_l"><long>18</long></arr><int name="id_i1">18</int><arr 
name="range_facet_l_dv"><long>18</long></arr><arr 
name="range_facet_i_dv"><int>18</int></arr><arr name="lowerfilt"><str>The quote 
red fox jumped over the lazy brown dogs.</str></arr><arr 
name="lowerfilt1and2"><str>The quote red fox jumped over the lazy brown 
dogs.</str></arr><arr name="multiDefault"><str>muLti-Default</str></arr><int 
name="intDefault">42</int><date 
name="timestamp">2018-02-06T12:58:12.229Z</date><int 
name="intDvoDefault">42</int></doc><doc><str name="id">19</str><arr 
name="range_facet_l"><long>19</long></arr><int name="id_i1">19</int><arr 
name="range_facet_l_dv"><long>19</long></arr><arr 
name="range_facet_i_dv"><int>19</int></arr><arr name="lowerfilt"><str>The hose 
red fox jumped over the lazy brown dogs.</str></arr><arr 
name="lowerfilt1and2"><str>The hose red fox jumped over the lazy brown 
dogs.</str></arr><arr name="multiDefault"><str>muLti-Default</str></arr><int 
name="intDefault">42</int><date 
name="timestamp">2018-02-06T12:58:12.230Z</date><int 
name="intDvoDefault">42</int></doc><doc><str name="id">13</str><arr 
name="range_facet_l"><long>13</long></arr><int name="id_i1">13</int><arr 
name="range_facet_l_dv"><long>13</long></arr><arr 
name="range_facet_i_dv"><int>13</int></arr><arr name="lowerfilt"><str>The quote 
red fox jumped over the lazy brown dogs.</str></arr><arr 
name="lowerfilt1and2"><str>The quote red fox jumped over the lazy brown 
dogs.</str></arr><arr name="multiDefault"><str>muLti-Default</str></arr><int 
name="intDefault">42</int><date 
name="timestamp">2018-02-06T12:58:12.081Z</date><int 
name="intDvoDefault">42</int></doc><doc><str name="id">30</str><arr 
name="range_facet_l"><long>30</long></arr><int name="id_i1">30</int><arr 
name="range_facet_l_dv"><long>30</long></arr><arr 
name="range_facet_i_dv"><int>30</int></arr><arr name="lowerfilt"><str>The quote 
red fox jumped over the lazy brown dogs.</str></arr><arr 
name="lowerfilt1and2"><str>The quote red fox jumped over the lazy brown 
dogs.</str><str>red green yellow</str></arr><arr name="lowerfilt1"><str>red 
green yellow</str></arr><arr 
name="multiDefault"><str>muLti-Default</str></arr><int 
name="intDefault">42</int><date 
name="timestamp">2018-02-06T12:58:13.663Z</date><int 
name="intDvoDefault">42</int></doc><doc><str name="id">31</str><arr 
name="range_facet_l"><long>31</long></arr><int name="id_i1">31</int><arr 
name="range_facet_l_dv"><long>31</long></arr><arr 
name="range_facet_i_dv"><int>31</int></arr><arr name="lowerfilt"><str>The fat 
red fox jumped over the lazy brown dogs.</str></arr><arr 
name="lowerfilt1and2"><str>The fat red fox jumped over the lazy brown 
dogs.</str><str>green blue yellow</str></arr><arr name="lowerfilt1"><str>green 
blue yellow</str></arr><arr 
name="multiDefault"><str>muLti-Default</str></arr><int 
name="intDefault">42</int><date 
name="timestamp">2018-02-06T12:58:13.678Z</date><int 
name="intDvoDefault">42</int></doc><doc><str name="id">20</str><arr 
name="range_facet_l"><long>20</long></arr><int name="id_i1">20</int><arr 
name="range_facet_l_dv"><long>20</long></arr><arr 
name="range_facet_i_dv"><int>20</int></arr><arr name="lowerfilt"><str>The quote 
red fox jumped over the lazy brown dogs.</str></arr><arr 
name="lowerfilt1and2"><str>The quote red fox jumped over the lazy brown 
dogs.</str></arr><arr name="multiDefault"><str>muLti-Default</str></arr><int 
name="intDefault">42</int><date 
name="timestamp">2018-02-06T12:58:13.410Z</date><int 
name="intDvoDefault">42</int></doc><doc><str name="id">21</str><arr 
name="range_facet_l"><long>21</long></arr><int name="id_i1">21</int><arr 
name="range_facet_l_dv"><long>21</long></arr><arr 
name="range_facet_i_dv"><int>21</int></arr><arr name="lowerfilt"><str>The court 
red fox jumped over the lazy brown dogs.</str></arr><arr 
name="lowerfilt1and2"><str>The court red fox jumped over the lazy brown 
dogs.</str></arr><arr name="multiDefault"><str>muLti-Default</str></arr><int 
name="intDefault">42</int><date 
name="timestamp">2018-02-06T12:58:13.411Z</date><int 
name="intDvoDefault">42</int></doc></result>
</response>

        request was:q={!mlt+qf%3Dlowerfilt}17
        at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:895)
        ... 41 more




Build Log:
[...truncated 13148 lines...]
   [junit4] Suite: org.apache.solr.cloud.MoveReplicaHDFSTest
   [junit4]   2> Creating dataDir: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.MoveReplicaHDFSTest_970C435508514221-001/init-core-data-001
   [junit4]   2> 736016 WARN  
(SUITE-MoveReplicaHDFSTest-seed#[970C435508514221]-worker) [    ] 
o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=24 numCloses=24
   [junit4]   2> 736137 INFO  
(SUITE-MoveReplicaHDFSTest-seed#[970C435508514221]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) 
w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 736139 INFO  
(SUITE-MoveReplicaHDFSTest-seed#[970C435508514221]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: 
@org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 736139 INFO  
(SUITE-MoveReplicaHDFSTest-seed#[970C435508514221]-worker) [    ] 
o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: 
test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 736140 INFO  
(SUITE-MoveReplicaHDFSTest-seed#[970C435508514221]-worker) [    ] 
o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.MoveReplicaHDFSTest_970C435508514221-001/tempDir-001
   [junit4]   2> 736140 INFO  
(SUITE-MoveReplicaHDFSTest-seed#[970C435508514221]-worker) [    ] 
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 736140 INFO  (Thread-713) [    ] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 736140 INFO  (Thread-713) [    ] o.a.s.c.ZkTestServer Starting 
server
   [junit4]   2> 736222 ERROR (Thread-713) [    ] o.a.z.s.ZooKeeperServer 
ZKShutdownHandler is not registered, so ZooKeeper server won't take any action 
on ERROR or SHUTDOWN server state changes
   [junit4]   2> 736240 INFO  
(SUITE-MoveReplicaHDFSTest-seed#[970C435508514221]-worker) [    ] 
o.a.s.c.ZkTestServer start zk server on port:45805
   [junit4]   2> 736288 INFO  (zkConnectionManagerCallback-2303-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 736303 INFO  (jetty-launcher-2300-thread-3) [    ] 
o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 
2017-11-22T01:27:37+04:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 736303 INFO  (jetty-launcher-2300-thread-2) [    ] 
o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 
2017-11-22T01:27:37+04:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 736321 INFO  (jetty-launcher-2300-thread-1) [    ] 
o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 
2017-11-22T01:27:37+04:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 736349 INFO  (jetty-launcher-2300-thread-3) [    ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 736349 INFO  (jetty-launcher-2300-thread-3) [    ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 736349 INFO  (jetty-launcher-2300-thread-3) [    ] 
o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 736349 INFO  (jetty-launcher-2300-thread-3) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@28208577{/solr,null,AVAILABLE}
   [junit4]   2> 736350 INFO  (jetty-launcher-2300-thread-3) [    ] 
o.e.j.s.AbstractConnector Started ServerConnector@38507dbc{SSL,[ssl, 
http/1.1]}{127.0.0.1:38320}
   [junit4]   2> 736350 INFO  (jetty-launcher-2300-thread-3) [    ] 
o.e.j.s.Server Started @796804ms
   [junit4]   2> 736350 INFO  (jetty-launcher-2300-thread-3) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=38320}
   [junit4]   2> 736350 ERROR (jetty-launcher-2300-thread-3) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 736350 INFO  (jetty-launcher-2300-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 
8.0.0
   [junit4]   2> 736350 INFO  (jetty-launcher-2300-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 736350 INFO  (jetty-launcher-2300-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 736350 INFO  (jetty-launcher-2300-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2018-02-06T12:44:33.950Z
   [junit4]   2> 736352 INFO  (jetty-launcher-2300-thread-2) [    ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 736352 INFO  (jetty-launcher-2300-thread-2) [    ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 736352 INFO  (jetty-launcher-2300-thread-2) [    ] 
o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 736352 INFO  (jetty-launcher-2300-thread-2) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@6dfb2449{/solr,null,AVAILABLE}
   [junit4]   2> 736366 INFO  (jetty-launcher-2300-thread-4) [    ] 
o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 
2017-11-22T01:27:37+04:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 736402 INFO  (jetty-launcher-2300-thread-4) [    ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 736402 INFO  (jetty-launcher-2300-thread-4) [    ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 736402 INFO  (jetty-launcher-2300-thread-4) [    ] 
o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 736406 INFO  (jetty-launcher-2300-thread-4) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@1f90b9c4{/solr,null,AVAILABLE}
   [junit4]   2> 736475 INFO  (jetty-launcher-2300-thread-1) [    ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 736475 INFO  (jetty-launcher-2300-thread-1) [    ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 736475 INFO  (jetty-launcher-2300-thread-1) [    ] 
o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 736476 INFO  (jetty-launcher-2300-thread-1) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@3e24dca6{/solr,null,AVAILABLE}
   [junit4]   2> 736476 INFO  (jetty-launcher-2300-thread-1) [    ] 
o.e.j.s.AbstractConnector Started ServerConnector@48d2a0d6{SSL,[ssl, 
http/1.1]}{127.0.0.1:42574}
   [junit4]   2> 736476 INFO  (jetty-launcher-2300-thread-1) [    ] 
o.e.j.s.Server Started @796931ms
   [junit4]   2> 736476 INFO  (jetty-launcher-2300-thread-1) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=42574}
   [junit4]   2> 736477 ERROR (jetty-launcher-2300-thread-1) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 736477 INFO  (jetty-launcher-2300-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 
8.0.0
   [junit4]   2> 736477 INFO  (jetty-launcher-2300-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 736477 INFO  (jetty-launcher-2300-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 736477 INFO  (jetty-launcher-2300-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2018-02-06T12:44:34.077Z
   [junit4]   2> 736564 INFO  (jetty-launcher-2300-thread-4) [    ] 
o.e.j.s.AbstractConnector Started ServerConnector@56a49ae6{SSL,[ssl, 
http/1.1]}{127.0.0.1:43978}
   [junit4]   2> 736564 INFO  (jetty-launcher-2300-thread-4) [    ] 
o.e.j.s.Server Started @797018ms
   [junit4]   2> 736564 INFO  (jetty-launcher-2300-thread-4) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=43978}
   [junit4]   2> 736564 ERROR (jetty-launcher-2300-thread-4) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 736564 INFO  (jetty-launcher-2300-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 
8.0.0
   [junit4]   2> 736564 INFO  (jetty-launcher-2300-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 736565 INFO  (jetty-launcher-2300-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 736565 INFO  (jetty-launcher-2300-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2018-02-06T12:44:34.165Z
   [junit4]   2> 736566 INFO  (zkConnectionManagerCallback-2309-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 736567 INFO  (jetty-launcher-2300-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 736597 INFO  (jetty-launcher-2300-thread-2) [    ] 
o.e.j.s.AbstractConnector Started ServerConnector@450a4e5d{SSL,[ssl, 
http/1.1]}{127.0.0.1:33908}
   [junit4]   2> 736597 INFO  (jetty-launcher-2300-thread-2) [    ] 
o.e.j.s.Server Started @797051ms
   [junit4]   2> 736597 INFO  (jetty-launcher-2300-thread-2) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=33908}
   [junit4]   2> 736597 ERROR (jetty-launcher-2300-thread-2) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 736597 INFO  (jetty-launcher-2300-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 
8.0.0
   [junit4]   2> 736597 INFO  (jetty-launcher-2300-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 736597 INFO  (jetty-launcher-2300-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 736597 INFO  (jetty-launcher-2300-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2018-02-06T12:44:34.197Z
   [junit4]   2> 736637 INFO  (zkConnectionManagerCallback-2305-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 736641 INFO  (zkConnectionManagerCallback-2307-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 736644 INFO  (zkConnectionManagerCallback-2311-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 736648 INFO  (jetty-launcher-2300-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 736662 INFO  (jetty-launcher-2300-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 736666 INFO  (jetty-launcher-2300-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 736714 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 
0x1000f6802cc0004, likely client has closed socket
   [junit4]   2> 736768 INFO  (jetty-launcher-2300-thread-4) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45805/solr
   [junit4]   2> 736825 INFO  (jetty-launcher-2300-thread-1) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45805/solr
   [junit4]   2> 736864 INFO  (zkConnectionManagerCallback-2315-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 736886 INFO  (jetty-launcher-2300-thread-2) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45805/solr
   [junit4]   2> 736887 INFO  (jetty-launcher-2300-thread-3) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45805/solr
   [junit4]   2> 736899 INFO  (zkConnectionManagerCallback-2323-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 736898 DEBUG (jetty-launcher-2300-thread-4) 
[n:127.0.0.1:43978_solr    ] o.a.s.c.ZkController Added new OnReconnect 
listener org.apache.solr.cloud.ZkController$$Lambda$245/1896885513@21d229b5
   [junit4]   2> 736900 INFO  (zkConnectionManagerCallback-2325-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 736914 INFO  (zkConnectionManagerCallback-2327-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 736915 DEBUG (jetty-launcher-2300-thread-1) 
[n:127.0.0.1:42574_solr    ] o.a.s.c.ZkController Added new OnReconnect 
listener org.apache.solr.cloud.ZkController$$Lambda$245/1896885513@623e12a1
   [junit4]   2> 736916 DEBUG (jetty-launcher-2300-thread-3) 
[n:127.0.0.1:38320_solr    ] o.a.s.c.ZkController Added new OnReconnect 
listener org.apache.solr.cloud.ZkController$$Lambda$245/1896885513@3905b0f8
   [junit4]   2> 736916 DEBUG (jetty-launcher-2300-thread-2) 
[n:127.0.0.1:33908_solr    ] o.a.s.c.ZkController Added new OnReconnect 
listener org.apache.solr.cloud.ZkController$$Lambda$245/1896885513@564e2067
   [junit4]   2> 736917 INFO  
(zkConnectionManagerCallback-2329-thread-1-processing-n:127.0.0.1:43978_solr) 
[n:127.0.0.1:43978_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 736927 INFO  
(zkConnectionManagerCallback-2331-thread-1-processing-n:127.0.0.1:42574_solr) 
[n:127.0.0.1:42574_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 736928 INFO  
(zkConnectionManagerCallback-2333-thread-1-processing-n:127.0.0.1:38320_solr) 
[n:127.0.0.1:38320_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 736928 INFO  
(zkConnectionManagerCallback-2335-thread-1-processing-n:127.0.0.1:33908_solr) 
[n:127.0.0.1:33908_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 737037 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: data: /solr/clusterprops.json
   [junit4]   2> 737038 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: children: /solr/live_nodes
   [junit4]   2> 737038 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: data: /solr/clusterstate.json
   [junit4]   2> 737039 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: children: /solr/collections
   [junit4]   2> 737052 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: create/delete: /solr/aliases.json
   [junit4]   2> 737082 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: create/delete: /solr/security.json
   [junit4]   2> 737109 INFO  (jetty-launcher-2300-thread-2) 
[n:127.0.0.1:33908_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 737110 DEBUG (jetty-launcher-2300-thread-2) 
[n:127.0.0.1:33908_solr    ] o.a.s.c.LeaderElector Joined leadership election 
with path: 
/overseer_elect/election/72074533435867148-127.0.0.1:33908_solr-n_0000000000
   [junit4]   2> 737112 INFO  (jetty-launcher-2300-thread-2) 
[n:127.0.0.1:33908_solr    ] o.a.s.c.OverseerElectionContext I am going to be 
the leader 127.0.0.1:33908_solr
   [junit4]   2> 737113 INFO  (jetty-launcher-2300-thread-2) 
[n:127.0.0.1:33908_solr    ] o.a.s.c.Overseer Overseer 
(id=72074533435867148-127.0.0.1:33908_solr-n_0000000000) starting
   [junit4]   2> 737162 DEBUG 
(OverseerCollectionConfigSetProcessor-72074533435867148-127.0.0.1:33908_solr-n_0000000000)
 [n:127.0.0.1:33908_solr    ] o.a.s.c.OverseerTaskProcessor Process current 
queue of overseer operations
   [junit4]   2> 737170 DEBUG 
(OverseerStateUpdate-72074533435867148-127.0.0.1:33908_solr-n_0000000000) 
[n:127.0.0.1:33908_solr    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 737178 INFO  (jetty-launcher-2300-thread-2) 
[n:127.0.0.1:33908_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:33908_solr
   [junit4]   2> 737179 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
fired: children: /solr/live_nodes
   [junit4]   2> 737194 DEBUG 
(OverseerAutoScalingTriggerThread-72074533435867148-127.0.0.1:33908_solr-n_0000000000)
 [n:127.0.0.1:33908_solr    ] o.a.s.c.a.OverseerTriggerThread Adding 
.autoAddReplicas trigger
   [junit4]   2> 737254 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: children: /solr/live_nodes
   [junit4]   2> 737254 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: data: /solr/autoscaling.json
   [junit4]   2> 737255 DEBUG 
(OverseerAutoScalingTriggerThread-72074533435867148-127.0.0.1:33908_solr-n_0000000000)
 [n:127.0.0.1:33908_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing 
/autoscaling.json with znode version 1
   [junit4]   2> 737256 DEBUG 
(OverseerAutoScalingTriggerThread-72074533435867148-127.0.0.1:33908_solr-n_0000000000)
 [n:127.0.0.1:33908_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: []
   [junit4]   2> 737281 DEBUG 
(OverseerAutoScalingTriggerThread-72074533435867148-127.0.0.1:33908_solr-n_0000000000)
 [n:127.0.0.1:33908_solr    ] o.a.s.c.a.OverseerTriggerThread Current 
znodeVersion 1, lastZnodeVersion -1
   [junit4]   2> 737281 DEBUG 
(OverseerAutoScalingTriggerThread-72074533435867148-127.0.0.1:33908_solr-n_0000000000)
 [n:127.0.0.1:33908_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger 
updates upto znodeVersion 1
   [junit4]   2> 737282 DEBUG 
(OverseerCollectionConfigSetProcessor-72074533435867148-127.0.0.1:33908_solr-n_0000000000)
 [n:127.0.0.1:33908_solr    ] o.a.s.c.OverseerTaskProcessor Cleaning up 
work-queue. #Running tasks: 0
   [junit4]   2> 737282 DEBUG 
(OverseerCollectionConfigSetProcessor-72074533435867148-127.0.0.1:33908_solr-n_0000000000)
 [n:127.0.0.1:33908_solr    ] o.a.s.c.OverseerTaskProcessor RunningTasks: []
   [junit4]   2> 737282 DEBUG 
(OverseerCollectionConfigSetProcessor-72074533435867148-127.0.0.1:33908_solr-n_0000000000)
 [n:127.0.0.1:33908_solr    ] o.a.s.c.OverseerTaskProcessor BlockedTasks: []
   [junit4]   2> 737282 DEBUG 
(OverseerCollectionConfigSetProcessor-72074533435867148-127.0.0.1:33908_solr-n_0000000000)
 [n:127.0.0.1:33908_solr    ] o.a.s.c.OverseerTaskProcessor CompletedTasks: []
   [junit4]   2> 737282 DEBUG 
(OverseerCollectionConfigSetProcessor-72074533435867148-127.0.0.1:33908_solr-n_0000000000)
 [n:127.0.0.1:33908_solr    ] o.a.s.c.OverseerTaskProcessor RunningZKTasks: []
   [junit4]   2> 737282 DEBUG 
(OverseerCollectionConfigSetProcessor-72074533435867148-127.0.0.1:33908_solr-n_0000000000)
 [n:127.0.0.1:33908_solr    ] o.a.s.c.OverseerTaskQueue Peeking for top 100 
elements. ExcludeSet: []
   [junit4]   2> 737282 INFO  
(OverseerStateUpdate-72074533435867148-127.0.0.1:33908_solr-n_0000000000) 
[n:127.0.0.1:33908_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 737283 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: children: /solr/overseer/collection-queue-work
   [junit4]   2> 737283 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: children: /solr/overseer/queue-work
   [junit4]   2> 737292 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: children: /solr/overseer/queue
   [junit4]   2> 737298 DEBUG 
(OverseerAutoScalingTriggerThread-72074533435867148-127.0.0.1:33908_solr-n_0000000000)
 [n:127.0.0.1:33908_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old 
nodeAdded markers
   [junit4]   2> 737298 DEBUG 
(OverseerAutoScalingTriggerThread-72074533435867148-127.0.0.1:33908_solr-n_0000000000)
 [n:127.0.0.1:33908_solr    ] o.a.s.c.a.OverseerTriggerThread Current 
znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 737299 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: children: /solr/autoscaling/events/.auto_add_replicas
   [junit4]   2> 737299 DEBUG (ScheduledTrigger-1389-thread-1) 
[n:127.0.0.1:33908_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: 
.auto_add_replicas with currently live nodes: 1
   [junit4]   2> 737536 INFO  (jetty-launcher-2300-thread-2) 
[n:127.0.0.1:33908_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_33908.solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@175ccc38
   [junit4]   2> 737575 INFO  (jetty-launcher-2300-thread-2) 
[n:127.0.0.1:33908_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_33908.solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@175ccc38
   [junit4]   2> 737575 INFO  (jetty-launcher-2300-thread-2) 
[n:127.0.0.1:33908_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_33908.solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@175ccc38
   [junit4]   2> 737576 INFO  (jetty-launcher-2300-thread-2) 
[n:127.0.0.1:33908_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.MoveReplicaHDFSTest_970C435508514221-001/tempDir-001/node3/.
   [junit4]   2> 737789 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: data: /solr/clusterprops.json
   [junit4]   2> 737790 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: children: /solr/live_nodes
   [junit4]   2> 737790 INFO  (jetty-launcher-2300-thread-1) 
[n:127.0.0.1:42574_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 737790 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: data: /solr/clusterstate.json
   [junit4]   2> 737791 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: children: /solr/collections
   [junit4]   2> 737791 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: create/delete: /solr/aliases.json
   [junit4]   2> 737804 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: create/delete: /solr/security.json
   [junit4]   2> 737805 INFO  (jetty-launcher-2300-thread-1) 
[n:127.0.0.1:42574_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 737806 DEBUG (jetty-launcher-2300-thread-1) 
[n:127.0.0.1:42574_solr    ] o.a.s.c.LeaderElector Joined leadership election 
with path: 
/overseer_elect/election/72074533435867146-127.0.0.1:42574_solr-n_0000000001
   [junit4]   2> 737806 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: data: 
/solr/overseer_elect/election/72074533435867148-127.0.0.1:33908_solr-n_0000000000
   [junit4]   2> 737806 DEBUG (jetty-launcher-2300-thread-1) 
[n:127.0.0.1:42574_solr    ] o.a.s.c.LeaderElector Watching path 
/overseer_elect/election/72074533435867148-127.0.0.1:33908_solr-n_0000000000 to 
know if I could be the leader
   [junit4]   2> 737806 DEBUG (jetty-launcher-2300-thread-1) 
[n:127.0.0.1:42574_solr    ] o.a.s.c.ZkController Publish 
node=127.0.0.1:42574_solr as DOWN
   [junit4]   2> 737807 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
fired: children: /solr/overseer/queue
   [junit4]   2> 737808 INFO  (jetty-launcher-2300-thread-1) 
[n:127.0.0.1:42574_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:42574_solr
   [junit4]   2> 737808 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
fired: children: /solr/live_nodes
   [junit4]   2> 737808 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
fired: children: /solr/live_nodes
   [junit4]   2> 737821 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: children: /solr/live_nodes
   [junit4]   2> 737856 INFO  
(zkCallback-2330-thread-1-processing-n:127.0.0.1:42574_solr) 
[n:127.0.0.1:42574_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (1) -> (2)
   [junit4]   2> 737870 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: children: /solr/overseer/queue
   [junit4]   2> 737870 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: children: /solr/live_nodes
   [junit4]   2> 737871 INFO  
(zkCallback-2334-thread-2-processing-n:127.0.0.1:33908_solr) 
[n:127.0.0.1:33908_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (1) -> (2)
   [junit4]   2> 737871 DEBUG 
(OverseerStateUpdate-72074533435867148-127.0.0.1:33908_solr-n_0000000000) 
[n:127.0.0.1:33908_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, 
message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:42574_solr"} current state version: 0
   [junit4]   2> 737871 DEBUG 
(OverseerStateUpdate-72074533435867148-127.0.0.1:33908_solr-n_0000000000) 
[n:127.0.0.1:33908_solr    ] o.a.s.c.o.NodeMutator DownNode state invoked for 
node: 127.0.0.1:42574_solr
   [junit4]   2> 737873 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: data: /solr/clusterprops.json
   [junit4]   2> 737886 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: children: /solr/live_nodes
   [junit4]   2> 737886 INFO  (jetty-launcher-2300-thread-4) 
[n:127.0.0.1:43978_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (2)
   [junit4]   2> 737886 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: data: /solr/clusterstate.json
   [junit4]   2> 737886 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: children: /solr/collections
   [junit4]   2> 737886 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: create/delete: /solr/aliases.json
   [junit4]   2> 737889 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: create/delete: /solr/security.json
   [junit4]   2> 737903 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: data: /solr/clusterprops.json
   [junit4]   2> 737903 INFO  (jetty-launcher-2300-thread-4) 
[n:127.0.0.1:43978_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 737903 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: children: /solr/live_nodes
   [junit4]   2> 737903 INFO  (jetty-launcher-2300-thread-3) 
[n:127.0.0.1:38320_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (2)
   [junit4]   2> 737903 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: data: /solr/clusterstate.json
   [junit4]   2> 737903 DEBUG (jetty-launcher-2300-thread-4) 
[n:127.0.0.1:43978_solr    ] o.a.s.c.LeaderElector Joined leadership election 
with path: 
/overseer_elect/election/72074533435867145-127.0.0.1:43978_solr-n_0000000002
   [junit4]   2> 737904 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: children: /solr/collections
   [junit4]   2> 737905 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: create/delete: /solr/aliases.json
   [junit4]   2> 737905 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: data: 
/solr/overseer_elect/election/72074533435867146-127.0.0.1:42574_solr-n_0000000001
   [junit4]   2> 737906 DEBUG (jetty-launcher-2300-thread-4) 
[n:127.0.0.1:43978_solr    ] o.a.s.c.LeaderElector Watching path 
/overseer_elect/election/72074533435867146-127.0.0.1:42574_solr-n_0000000001 to 
know if I could be the leader
   [junit4]   2> 737914 DEBUG (jetty-launcher-2300-thread-4) 
[n:127.0.0.1:43978_solr    ] o.a.s.c.ZkController Publish 
node=127.0.0.1:43978_solr as DOWN
   [junit4]   2> 737919 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: create/delete: /solr/security.json
   [junit4]   2> 737919 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
fired: children: /solr/overseer/queue
   [junit4]   2> 737919 INFO  (jetty-launcher-2300-thread-4) 
[n:127.0.0.1:43978_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:43978_solr
   [junit4]   2> 737920 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
fired: children: /solr/live_nodes
   [junit4]   2> 737920 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
fired: children: /solr/live_nodes
   [junit4]   2> 737920 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
fired: children: /solr/live_nodes
   [junit4]   2> 737920 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
fired: children: /solr/live_nodes
   [junit4]   2> 737935 INFO  (jetty-launcher-2300-thread-3) 
[n:127.0.0.1:38320_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 737955 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: children: /solr/live_nodes
   [junit4]   2> 737955 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: children: /solr/live_nodes
   [junit4]   2> 737955 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: children: /solr/live_nodes
   [junit4]   2> 737955 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: children: /solr/live_nodes
   [junit4]   2> 737955 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: children: /solr/overseer/queue
   [junit4]   2> 737955 INFO  
(zkCallback-2334-thread-1-processing-n:127.0.0.1:33908_solr) 
[n:127.0.0.1:33908_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (2) -> (3)
   [junit4]   2> 737959 INFO  
(zkCallback-2330-thread-1-processing-n:127.0.0.1:42574_solr) 
[n:127.0.0.1:42574_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (2) -> (3)
   [junit4]   2> 737976 INFO  
(zkCallback-2328-thread-1-processing-n:127.0.0.1:43978_solr) 
[n:127.0.0.1:43978_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (2) -> (3)
   [junit4]   2> 737976 INFO  
(zkCallback-2332-thread-1-processing-n:127.0.0.1:38320_solr) 
[n:127.0.0.1:38320_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (2) -> (3)
   [junit4]   2> 737980 DEBUG (jetty-launcher-2300-thread-3) 
[n:127.0.0.1:38320_solr    ] o.a.s.c.LeaderElector Joined leadership election 
with path: 
/overseer_elect/election/72074533435867147-127.0.0.1:38320_solr-n_0000000003
   [junit4]   2> 737980 DEBUG 
(OverseerStateUpdate-72074533435867148-127.0.0.1:33908_solr-n_0000000000) 
[n:127.0.0.1:33908_solr    ] o.a.s.c.Overseer processMessage: queueSize: 2, 
message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:43978_solr"} current state version: 0
   [junit4]   2> 737980 DEBUG 
(OverseerStateUpdate-72074533435867148-127.0.0.1:33908_solr-n_0000000000) 
[n:127.0.0.1:33908_solr    ] o.a.s.c.o.NodeMutator DownNode state invoked for 
node: 127.0.0.1:43978_solr
   [junit4]   2> 737981 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: data: 
/solr/overseer_elect/election/72074533435867145-127.0.0.1:43978_solr-n_0000000002
   [junit4]   2> 737981 DEBUG (jetty-launcher-2300-thread-3) 
[n:127.0.0.1:38320_solr    ] o.a.s.c.LeaderElector Watching path 
/overseer_elect/election/72074533435867145-127.0.0.1:43978_solr-n_0000000002 to 
know if I could be the leader
   [junit4]   2> 737981 DEBUG (jetty-launcher-2300-thread-3) 
[n:127.0.0.1:38320_solr    ] o.a.s.c.ZkController Publish 
node=127.0.0.1:38320_solr as DOWN
   [junit4]   2> 737982 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
fired: children: /solr/overseer/queue
   [junit4]   2> 737982 INFO  (jetty-launcher-2300-thread-3) 
[n:127.0.0.1:38320_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:38320_solr
   [junit4]   2> 737982 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
fired: children: /solr/live_nodes
   [junit4]   2> 737982 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
fired: children: /solr/live_nodes
   [junit4]   2> 737982 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
fired: children: /solr/live_nodes
   [junit4]   2> 737982 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
fired: children: /solr/live_nodes
   [junit4]   2> 737983 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: children: /solr/live_nodes
   [junit4]   2> 738004 INFO  
(zkCallback-2330-thread-1-processing-n:127.0.0.1:42574_solr) 
[n:127.0.0.1:42574_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (3) -> (4)
   [junit4]   2> 738007 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: children: /solr/overseer/queue
   [junit4]   2> 738007 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: children: /solr/live_nodes
   [junit4]   2> 738007 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: children: /solr/live_nodes
   [junit4]   2> 738007 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: children: /solr/live_nodes
   [junit4]   2> 738007 INFO  
(zkCallback-2332-thread-1-processing-n:127.0.0.1:38320_solr) 
[n:127.0.0.1:38320_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (3) -> (4)
   [junit4]   2> 738007 INFO  
(zkCallback-2334-thread-2-processing-n:127.0.0.1:33908_solr) 
[n:127.0.0.1:33908_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (3) -> (4)
   [junit4]   2> 738007 INFO  
(zkCallback-2328-thread-1-processing-n:127.0.0.1:43978_solr) 
[n:127.0.0.1:43978_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (3) -> (4)
   [junit4]   2> 738007 DEBUG 
(OverseerStateUpdate-72074533435867148-127.0.0.1:33908_solr-n_0000000000) 
[n:127.0.0.1:33908_solr    ] o.a.s.c.Overseer processMessage: queueSize: 3, 
message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:38320_solr"} current state version: 0
   [junit4]   2> 738007 DEBUG 
(OverseerStateUpdate-72074533435867148-127.0.0.1:33908_solr-n_0000000000) 
[n:127.0.0.1:33908_solr    ] o.a.s.c.o.NodeMutator DownNode state invoked for 
node: 127.0.0.1:38320_solr
   [junit4]   2> 738108 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
fired: children: /solr/overseer/queue
   [junit4]   2> 738109 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: children: /solr/overseer/queue
   [junit4]   2> 738142 INFO  (jetty-launcher-2300-thread-1) 
[n:127.0.0.1:42574_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_42574.solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@175ccc38
   [junit4]   2> 738162 INFO  (jetty-launcher-2300-thread-1) 
[n:127.0.0.1:42574_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_42574.solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@175ccc38
   [junit4]   2> 738162 INFO  (jetty-launcher-2300-thread-1) 
[n:127.0.0.1:42574_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_42574.solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@175ccc38
   [junit4]   2> 738188 INFO  (jetty-launcher-2300-thread-1) 
[n:127.0.0.1:42574_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.MoveReplicaHDFSTest_970C435508514221-001/tempDir-001/node1/.
   [junit4]   2> 738194 INFO  (jetty-launcher-2300-thread-4) 
[n:127.0.0.1:43978_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_43978.solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@175ccc38
   [junit4]   2> 738251 INFO  (jetty-launcher-2300-thread-4) 
[n:127.0.0.1:43978_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_43978.solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@175ccc38
   [junit4]   2> 738259 INFO  (jetty-launcher-2300-thread-4) 
[n:127.0.0.1:43978_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_43978.solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@175ccc38
   [junit4]   2> 738293 INFO  (jetty-launcher-2300-thread-4) 
[n:127.0.0.1:43978_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.MoveReplicaHDFSTest_970C435508514221-001/tempDir-001/node4/.
   [junit4]   2> 738330 DEBUG (ScheduledTrigger-1389-thread-1) 
[n:127.0.0.1:33908_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: 
.auto_add_replicas with currently live nodes: 4
   [junit4]   2> 738331 INFO  (jetty-launcher-2300-thread-3) 
[n:127.0.0.1:38320_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_38320.solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@175ccc38
   [junit4]   2> 738367 INFO  (jetty-launcher-2300-thread-3) 
[n:127.0.0.1:38320_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_38320.solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@175ccc38
   [junit4]   2> 738367 INFO  (jetty-launcher-2300-thread-3) 
[n:127.0.0.1:38320_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_38320.solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@175ccc38
   [junit4]   2> 738368 INFO  (jetty-launcher-2300-thread-3) 
[n:127.0.0.1:38320_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.MoveReplicaHDFSTest_970C435508514221-001/tempDir-001/node2/.
   [junit4]   2> 738866 INFO  (zkConnectionManagerCallback-2345-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 738951 INFO  (zkConnectionManagerCallback-2349-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 738979 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: data: /solr/clusterprops.json
   [junit4]   2> 738979 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: children: /solr/live_nodes
   [junit4]   2> 738979 INFO  
(SUITE-MoveReplicaHDFSTest-seed#[970C435508514221]-worker) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 738980 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: data: /solr/clusterstate.json
   [junit4]   2> 738992 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: children: /solr/collections
   [junit4]   2> 738992 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: create/delete: /solr/aliases.json
   [junit4]   2> 738993 INFO  
(SUITE-MoveReplicaHDFSTest-seed#[970C435508514221]-worker) [    ] 
o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:45805/solr ready
   [junit4]   2> 739099 INFO  (qtp1846962903-6057) [n:127.0.0.1:42574_solr    ] 
o.a.s.h.a.CollectionsHandler Invoked Collection Action :overseerstatus with 
params action=OVERSEERSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 739100 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
fired: children: /solr/overseer/collection-queue-work
   [junit4]   2> 739100 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: create/delete: /solr/overseer/collection-queue-work/qnr-0000000000
   [junit4]   2> 739100 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: children: /solr/overseer/collection-queue-work
   [junit4]   2> 739100 DEBUG 
(OverseerCollectionConfigSetProcessor-72074533435867148-127.0.0.1:33908_solr-n_0000000000)
 [n:127.0.0.1:33908_solr    ] o.a.s.c.ZkDistributedQueue Found child node with 
improper name: qnr-0000000000
   [junit4]   2> 739100 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
fired: children: /solr/overseer/collection-queue-work
   [junit4]   2> 739101 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: children: /solr/overseer/collection-queue-work
   [junit4]   2> 739101 DEBUG 
(OverseerCollectionConfigSetProcessor-72074533435867148-127.0.0.1:33908_solr-n_0000000000)
 [n:127.0.0.1:33908_solr    ] o.a.s.c.ZkDistributedQueue Found child node with 
improper name: qnr-0000000000
   [junit4]   2> 739101 DEBUG 
(OverseerCollectionConfigSetProcessor-72074533435867148-127.0.0.1:33908_solr-n_0000000000)
 [n:127.0.0.1:33908_solr    ] o.a.s.c.OverseerTaskQueue Returning topN 
elements: [/overseer/collection-queue-work/qn-0000000000, ]
   [junit4]   2> 739101 DEBUG 
(OverseerCollectionConfigSetProcessor-72074533435867148-127.0.0.1:33908_solr-n_0000000000)
 [n:127.0.0.1:33908_solr    ] o.a.s.c.OverseerTaskProcessor Got 1 tasks from 
work-queue : [[org.apache.solr.cloud.OverseerTaskQueue$QueueEvent@2e684ecf]]
   [junit4]   2> 739101 DEBUG 
(OverseerCollectionConfigSetProcessor-72074533435867148-127.0.0.1:33908_solr-n_0000000000)
 [n:127.0.0.1:33908_solr    ] o.a.s.c.OverseerTaskProcessor Marked task 
[/overseer/collection-queue-work/qn-0000000000] as running
   [junit4]   2> 739101 DEBUG 
(OverseerCollectionConfigSetProcessor-72074533435867148-127.0.0.1:33908_solr-n_0000000000)
 [n:127.0.0.1:33908_solr    ] o.a.s.c.OverseerTaskProcessor Overseer Collection 
Message Handler: Get the message 
id:/overseer/collection-queue-work/qn-0000000000 
message:{"operation":"overseerstatus"}
   [junit4]   2> 739110 DEBUG 
(OverseerCollectionConfigSetProcessor-72074533435867148-127.0.0.1:33908_solr-n_0000000000)
 [n:127.0.0.1:33908_solr    ] o.a.s.c.OverseerTaskProcessor Cleaning up 
work-queue. #Running tasks: 1
   [junit4]   2> 739110 DEBUG 
(OverseerCollectionConfigSetProcessor-72074533435867148-127.0.0.1:33908_solr-n_0000000000)
 [n:127.0.0.1:33908_solr    ] o.a.s.c.OverseerTaskProcessor RunningTasks: 
[/overseer/collection-queue-work/qn-0000000000]
   [junit4]   2> 739110 DEBUG 
(OverseerCollectionConfigSetProcessor-72074533435867148-127.0.0.1:33908_solr-n_0000000000)
 [n:127.0.0.1:33908_solr    ] o.a.s.c.OverseerTaskProcessor BlockedTasks: []
   [junit4]   2> 739110 DEBUG 
(OverseerCollectionConfigSetProcessor-72074533435867148-127.0.0.1:33908_solr-n_0000000000)
 [n:127.0.0.1:33908_solr    ] o.a.s.c.OverseerTaskProcessor CompletedTasks: []
   [junit4]   2> 739110 DEBUG 
(OverseerCollectionConfigSetProcessor-72074533435867148-127.0.0.1:33908_solr-n_0000000000)
 [n:127.0.0.1:33908_solr    ] o.a.s.c.OverseerTaskProcessor RunningZKTasks: 
[/overseer/collection-queue-work/qn-0000000000]
   [junit4]   2> 739111 DEBUG 
(OverseerCollectionConfigSetProcessor-72074533435867148-127.0.0.1:33908_solr-n_0000000000)
 [n:127.0.0.1:33908_solr    ] o.a.s.c.OverseerTaskQueue Peeking for top 99 
elements. ExcludeSet: [/overseer/collection-queue-work/qn-0000000000],[]
   [junit4]   2> 739110 DEBUG 
(OverseerThreadFactory-1391-thread-1-processing-n:127.0.0.1:33908_solr) 
[n:127.0.0.1:33908_solr    ] o.a.s.c.OverseerTaskProcessor Runner processing 
/overseer/collection-queue-work/qn-0000000000
   [junit4]   2> 739111 DEBUG 
(OverseerThreadFactory-1391-thread-1-processing-n:127.0.0.1:33908_solr) 
[n:127.0.0.1:33908_solr    ] o.a.s.c.a.c.OverseerCollectionMessageHandler 
OverseerCollectionMessageHandler.processMessage : overseerstatus , 
{"operation":"overseerstatus"}
   [junit4]   2> 739132 DEBUG 
(OverseerThreadFactory-1391-thread-1-processing-n:127.0.0.1:33908_solr) 
[n:127.0.0.1:33908_solr    ] o.a.s.c.OverseerTaskProcessor Completed 
task:[/overseer/collection-queue-work/qn-0000000000]
   [junit4]   2> 739133 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
fired: data: /solr/overseer/collection-queue-work/qnr-0000000000
   [junit4]   2> 739133 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
fired: children: /solr/overseer/collection-queue-work
   [junit4]   2> 739133 DEBUG 
(OverseerThreadFactory-1391-thread-1-processing-n:127.0.0.1:33908_solr) 
[n:127.0.0.1:33908_solr    ] o.a.s.c.OverseerTaskProcessor Marked task 
[/overseer/collection-queue-work/qn-0000000000] as completed.
   [junit4]   2> 739133 DEBUG 
(OverseerThreadFactory-1391-thread-1-processing-n:127.0.0.1:33908_solr) 
[n:127.0.0.1:33908_solr    ] o.a.s.c.OverseerTaskProcessor RunningTasks: []
   [junit4]   2> 739133 DEBUG 
(OverseerThreadFactory-1391-thread-1-processing-n:127.0.0.1:33908_solr) 
[n:127.0.0.1:33908_solr    ] o.a.s.c.OverseerTaskProcessor BlockedTasks: []
   [junit4]   2> 739133 DEBUG 
(OverseerThreadFactory-1391-thread-1-processing-n:127.0.0.1:33908_solr) 
[n:127.0.0.1:33908_solr    ] o.a.s.c.OverseerTaskProcessor CompletedTasks: 
[/overseer/collection-queue-work/qn-0000000000]
   [junit4]   2> 739133 DEBUG 
(OverseerThreadFactory-1391-thread-1-processing-n:127.0.0.1:33908_solr) 
[n:127.0.0.1:33908_solr    ] o.a.s.c.OverseerTaskProcessor RunningZKTasks: 
[/overseer/collection-queue-work/qn-0000000000]
   [junit4]   2> 739134 DEBUG 
(OverseerThreadFactory-1391-thread-1-processing-n:127.0.0.1:33908_solr) 
[n:127.0.0.1:33908_solr    ] o.a.s.c.OverseerTaskProcessor Overseer Collection 
Message Handler: Message id:/overseer/collection-queue-work/qn-0000000000 
complete, 
response:{leader=127.0.0.1:33908_solr,overseer_queue_size=0,overseer_work_queue_size=0,overseer_collection_queue_size=2,overseer_operations={am_i_leader={requests=3,errors=0,avgRequestsPerSecond=1.5229588274370822,5minRateRequestsPerSecond=0.0,15minRateRequestsPerSecond=0.0,avgTimePerRequest=5.987176507014329,medianRequestTime=8.421375,75thPcRequestTime=9.382245,95thPcRequestTime=9.382245,99thPcRequestTime=9.382245,999thPcRequestTime=9.382245},downnode={requests=3,errors=0,avgRequestsPerSecond=2.3817537446831483,5minRateRequestsPerSecond=0.0,15minRateRequestsPerSecond=0.0,avgTimePerRequest=0.047644,medianRequestTime=0.047307,75thPcRequestTime=0.049874,95thPcRequestTime=0.049874,99thPcRequestTime=0.049874,999thPcRequestTime=0.049874}},collection_operations={am_i_leader={requests=3,errors=0,avgRequestsPerSecond=1.5244189956220933,5minRateRequestsPerSecond=0.0,15minRateRequestsPerSecond=0.0,avgTimePerRequest=13.222373371450518,medianRequestTime=16.2762,75thPcRequestTime=23.301008,95thPcRequestTime=23.301008,99thPcRequestTime=23.301008,999thPcRequestTime=23.301008},overseerstatus={requests=0,errors=0,avgRequestsPerSecond=0.0,5minRateRequestsPerSecond=0.0,15minRateRequestsPerSecond=0.0,avgTimePerRequest=0.0,medianRequestTime=0.0,75thPcRequestTime=0.0,95thPcRequestTime=0.0,99thPcRequestTime=0.0,999thPcRequestTime=0.0}},overseer_queue={},overseer_internal_queue={peek={avgRequestsPerSecond=0.5412518171075146,5minRateRequestsPerSecond=0.0,15minRateRequestsPerSecond=0.0,avgTimePerRequest=0.378051,medianRequestTime=0.378051,75thPcRequestTime=0.378051,95thPcRequestTime=0.378051,99thPcRequestTime=0.378051,999thPcRequestTime=0.378051}},collection_queue={peektopn_wait2000={avgRequestsPerSecond=0.5411128560112836,5minRateRequestsPerSecond=0.0,15minRateRequestsPerSecond=0.0,avgTimePerRequest=1818.642279,medianRequestTime=1818.642279,75thPcRequestTime=1818.642279,95thPcRequestTime=1818.642279,99thPcRequestTime=1818.642279,999thPcRequestTime=1818.642279}}}
   [junit4]   2> 739147 DEBUG 
(zkCallback-2330-thread-1-processing-n:127.0.0.1:42574_solr) 
[n:127.0.0.1:42574_solr    ] o.a.s.c.OverseerTaskQueue NodeDataChanged fired on 
path /overseer/collection-queue-work/qnr-0000000000 state SyncConnected 
latchEventType null
   [junit4]   2> 739147 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: children: /solr/overseer/collection-queue-work
   [junit4]   2> 739147 DEBUG 
(OverseerCollectionConfigSetProcessor-72074533435867148-127.0.0.1:33908_solr-n_0000000000)
 [n:127.0.0.1:33908_solr    ] o.a.s.c.ZkDistributedQueue Found child node with 
improper name: qnr-0000000000
   [junit4]   2> 739148 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
fired: children: /solr/overseer/collection-queue-work
   [junit4]   2> 739148 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch 
added: children: /solr/overseer/collection-queue-work
   [junit4]   2> 739149 INFO  (qtp1846962903-6057) [n:127.0.0.1:42574_solr    ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={action=OVERSEERSTATUS&wt=javabin&version=2} status=0 QTime=50
   [junit4]   2> 739378 DEBUG (ScheduledTrigger-1389-thread-1) 
[n:127.0.0.1:33908_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: 
.auto_add_replicas with currently live nodes: 4
   [junit4]   2> 740464 DEBUG (ScheduledTrigger-1389-thread-3) 
[n:127.0.0.1:33908_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: 
.auto_add_replicas with currently live nodes: 4
   [junit4]   2> 741114 DEBUG 
(OverseerCollectionConfigSetProcessor-72074533435867148-127.0.0.1:33908_solr-n_0000000000)
 [n:127.0.0.1:33908_solr    ] o.a.s.c.OverseerTaskProcessor Got 0 tasks from 
work-queue : [[]]
   [junit4]   2> 741166 DEBUG 
(OverseerCollectionConfigSetProcessor-72074533435867148-127.0.0.1:33908_solr-n_0000000000)
 [n:127.0.0.1:33908_solr    ] o.a.s.c.OverseerTaskProcessor Cleaning up 
work-queue. #Running tasks: 0
   [junit4]   2> 741179 INFO  
(OverseerCollectionConfigSetProcessor-72074533435867148-127.0.0.1:33908_solr-n_0000000000)
 [n:127.0.0.1:33908_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: 
/overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may 
have disconnected from ZooKeeper
   [junit4]   2> 741180 DEBUG 
(OverseerCollectionConfigSetProcessor-72074533435867148-127.0.0.1:33908_solr-n_0000000000)
 [n:127.0.0.1:33908_solr    ] o.a.s.c.OverseerTaskProcessor RunningTasks: []
   [junit4]   2> 741180 DEBUG 
(OverseerCollectionConfigSetProcessor-72074533435867148-127.0.0.1:33908_solr-n_0000000000)
 [n:127.0.0.1:33908_solr    ] o.a.s.c.OverseerTaskProcessor BlockedTasks: []
   [junit4]   2> 741180 DEBUG 
(OverseerCollectionConfigSetProcessor-72074533435867148-127.0.0.1:33908_solr-n_0000000000)
 [n:127.0.0.1:33908_solr    ] o.a.s.c.OverseerTaskProcessor CompletedTasks: []
   [junit4]   2> 741180 DEBUG 
(OverseerCollectionConfigSetProcessor-72074533435867148-127.0.0.1:33908_solr-n_0000000000)
 [n:127.0.0.1:33908_solr    ] o.a.s.c.OverseerTaskProcessor RunningZKTasks: []
   [junit4]   2> 741180 DEBUG 
(OverseerCollectionConfigSetProcessor-72074533435867148-127.0.0.1:33908_solr-n_0000000000)
 [n:127.0.0.1:33908_solr    ] o.a.s.c.OverseerTaskQueue Peeking for top 100 
elements. ExcludeSet: []
   [junit4]   2> 741472 DEBUG (ScheduledTrigger-1389-thread-3) 
[n:127.0.0.1:33908_solr   

[...truncated too long message...]

er Closing metric reporters for registry=solr.core.collection1, tag=1931642282
   [junit4]   2> 58809 INFO  
(SUITE-TestFieldLengthFeature-seed#[A539D143B4BF2303]-worker) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@1de82011{/solr,null,UNAVAILABLE}
   [junit4]   2> 58809 INFO  
(SUITE-TestFieldLengthFeature-seed#[A539D143B4BF2303]-worker) [    ] 
o.e.j.s.session Stopped scavenging
   [junit4]   2> 88810 WARN  
(SUITE-TestFieldLengthFeature-seed#[A539D143B4BF2303]-worker) [    ] 
o.e.j.u.t.QueuedThreadPool 
QueuedThreadPool@qtp1192070498{STOPPING,8<=8<=10000,i=0,q=1} Couldn't stop 
Thread[qtp1192070498-88,5,TGRP-TestFieldLengthFeature]
   [junit4]   2> 88826 INFO  
(SUITE-TestFieldLengthFeature-seed#[A539D143B4BF2303]-worker) [    ] 
o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> 88826 INFO  
(SUITE-TestFieldLengthFeature-seed#[A539D143B4BF2303]-worker) [    ] 
o.a.s.c.CoreContainer Shutting down CoreContainer instance=280502451
   [junit4]   2> 88826 INFO  
(SUITE-TestFieldLengthFeature-seed#[A539D143B4BF2303]-worker) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, 
tag=null
   [junit4]   2> 88827 INFO  
(SUITE-TestFieldLengthFeature-seed#[A539D143B4BF2303]-worker) [    ] 
o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@54740f6f: rootName = null, 
domain = solr.node, service url = null, agent id = null] for registry solr.node 
/ null
   [junit4]   2> 88827 INFO  
(SUITE-TestFieldLengthFeature-seed#[A539D143B4BF2303]-worker) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, 
tag=null
   [junit4]   2> 88827 INFO  
(SUITE-TestFieldLengthFeature-seed#[A539D143B4BF2303]-worker) [    ] 
o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@1bd96aee: rootName = null, 
domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / 
null
   [junit4]   2> 88827 INFO  
(SUITE-TestFieldLengthFeature-seed#[A539D143B4BF2303]-worker) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, 
tag=null
   [junit4]   2> 88827 INFO  
(SUITE-TestFieldLengthFeature-seed#[A539D143B4BF2303]-worker) [    ] 
o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@2c5aa4e8: rootName = null, 
domain = solr.jetty, service url = null, agent id = null] for registry 
solr.jetty / null
   [junit4]   2> 88843 INFO  (coreCloseExecutor-77-thread-1) [    
x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore 
org.apache.solr.core.SolrCore@4388290d
   [junit4]   2> 88843 INFO  (coreCloseExecutor-77-thread-1) [    
x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for 
registry=solr.core.collection1, tag=1132996877
   [junit4]   2> 88843 INFO  (coreCloseExecutor-77-thread-1) [    
x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@4431db5e: rootName = null, 
domain = solr.core.collection1, service url = null, agent id = null] for 
registry solr.core.collection1 / null
   [junit4]   2> 88918 INFO  
(SUITE-TestFieldLengthFeature-seed#[A539D143B4BF2303]-worker) [    ] 
o.a.s.SolrTestCaseJ4 ------------------------------------------------------- 
Done waiting for tracked resources to be released
   [junit4]   2> Feb 06, 2018 3:20:49 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked 
thread(s).
   [junit4]   2> Feb 06, 2018 3:21:09 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at 
org.apache.solr.ltr.feature.TestFieldLengthFeature: 
   [junit4]   2>    1) Thread[id=88, name=qtp1192070498-88, 
state=TIMED_WAITING, group=TGRP-TestFieldLengthFeature]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]   2>         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
   [junit4]   2>         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
   [junit4]   2>         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
   [junit4]   2>         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
   [junit4]   2>         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Feb 06, 2018 3:21:09 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=88, name=qtp1192070498-88, 
state=TIMED_WAITING, group=TGRP-TestFieldLengthFeature]
   [junit4]   2> Feb 06, 2018 3:21:12 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> SEVERE: There are still zombie threads that couldn't be 
terminated:
   [junit4]   2>    1) Thread[id=88, name=qtp1192070498-88, 
state=TIMED_WAITING, group=TGRP-TestFieldLengthFeature]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]   2>         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
   [junit4]   2>         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
   [junit4]   2>         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
   [junit4]   2>         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
   [junit4]   2>         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: test params are: 
codec=HighCompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=HIGH_COMPRESSION,
 chunkSize=7, maxDocsPerChunk=7, blockSize=2), 
termVectorsFormat=CompressingTermVectorsFormat(compressionMode=HIGH_COMPRESSION,
 chunkSize=7, blockSize=2)), 
sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@43be5cfb),
 locale=sr-Latn, timezone=Africa/Accra
   [junit4]   2> NOTE: Linux 4.4.0-112-generic amd64/Oracle Corporation 
1.8.0_152 (64-bit)/cpus=4,threads=2,free=243112920,total=303562752
   [junit4]   2> NOTE: All tests run in this JVM: [TestExternalValueFeatures, 
TestRankingFeature, TestFieldLengthFeature]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 
'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=TestFieldLengthFeature -Dtests.seed=A539D143B4BF2303 
-Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt
 -Dtests.locale=sr-Latn -Dtests.timezone=Africa/Accra -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J0 | TestFieldLengthFeature (suite) <<<
   [junit4]    > Throwable #1: 
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE 
scope at org.apache.solr.ltr.feature.TestFieldLengthFeature: 
   [junit4]    >    1) Thread[id=88, name=qtp1192070498-88, 
state=TIMED_WAITING, group=TGRP-TestFieldLengthFeature]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([A539D143B4BF2303]:0)Throwable #2: 
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie 
threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=88, name=qtp1192070498-88, 
state=TIMED_WAITING, group=TGRP-TestFieldLengthFeature]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([A539D143B4BF2303]:0)
   [junit4] Completed [28/35 (1!)] on J0 in 59.85s, 3 tests, 2 errors <<< 
FAILURES!

[...truncated 40713 lines...]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to