Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/70/

10 tests failed.
FAILED:  org.apache.lucene.index.TestIndexWriterDelete.testUpdatesOnDiskFull

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([6161F631F6B80D4A]:0)


FAILED:  junit.framework.TestSuite.org.apache.lucene.index.TestIndexWriterDelete

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([6161F631F6B80D4A]:0)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsWriteToMultipleCollectionsTest

Error Message:


Stack Trace:
java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([CC3F0BBD3EEA3BFC]:0)
        at 
org.apache.lucene.util.TestRuleRestoreSystemProperties.before(TestRuleRestoreSystemProperties.java:52)
        at 
com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.before(TestRuleAdapter.java:26)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:35)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  
junit.framework.TestSuite.org.apache.solr.handler.admin.PropertiesRequestHandlerTest

Error Message:


Stack Trace:
java.lang.AssertionError
        at 
org.apache.lucene.util.TestRuleTemporaryFilesCleanup.before(TestRuleTemporaryFilesCleanup.java:112)
        at 
com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.before(TestRuleAdapter.java:26)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:35)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  
junit.framework.TestSuite.org.apache.solr.handler.admin.PropertiesRequestHandlerTest

Error Message:
14 threads leaked from SUITE scope at 
org.apache.solr.handler.admin.PropertiesRequestHandlerTest:     1) 
Thread[id=3818, name=coreZkRegister-894-thread-1, state=TIMED_WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at 
java.lang.Object.wait(Native Method)         at 
java.lang.Object.wait(Object.java:502)         at 
org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1406)         at 
org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1097)         at 
org.apache.solr.common.cloud.SolrZkClient.lambda$exists$3(SolrZkClient.java:308)
         at 
org.apache.solr.common.cloud.SolrZkClient$$Lambda$201/1712956663.execute(Unknown
 Source)         at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
         at 
org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:308)         
at 
org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:93)  
       at org.apache.solr.cloud.LeaderElector.setup(LeaderElector.java:367)     
    at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1288)  
       at org.apache.solr.cloud.ZkController.register(ZkController.java:1060)   
      at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:256)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$117/1778849115.run(Unknown
 Source)         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
        at java.lang.Thread.run(Thread.java:748)    2) Thread[id=3804, 
name=zkCallback-998-thread-30, state=TIMED_WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at 
java.lang.Object.wait(Native Method)         at 
java.lang.Object.wait(Object.java:502)         at 
org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1406)         at 
org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:781)         at 
org.apache.solr.common.cloud.SolrZkClient.lambda$create$7(SolrZkClient.java:357)
         at 
org.apache.solr.common.cloud.SolrZkClient$$Lambda$205/1874600898.execute(Unknown
 Source)         at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
         at 
org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:357)         
at org.apache.solr.cloud.Overseer.createOverseerNode(Overseer.java:765)         
at org.apache.solr.cloud.Overseer.getCollectionQueue(Overseer.java:714)         
at 
org.apache.solr.cloud.OverseerCollectionConfigSetProcessor.<init>(OverseerCollectionConfigSetProcessor.java:48)
         at org.apache.solr.cloud.Overseer.start(Overseer.java:523)         at 
org.apache.solr.cloud.OverseerElectionContext.runLeaderProcess(ElectionContext.java:799)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)    
     at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:56)   
      at 
org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:348)
         at 
org.apache.solr.common.cloud.SolrZkClient$1.lambda$process$0(SolrZkClient.java:262)
         at 
org.apache.solr.common.cloud.SolrZkClient$1$$Lambda$191/209459966.run(Unknown 
Source)         at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)         
at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$117/1778849115.run(Unknown
 Source)         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
        at java.lang.Thread.run(Thread.java:748)    3) Thread[id=3761, 
name=zkCallback-998-thread-5-processing-n:127.0.0.1:42259_-SendThread(127.0.0.1:44052),
 state=TERMINATED, group={null group}]         at java.lang.Thread.sleep(Native 
Method)         at 
org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:230)  
       at 
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1246)        
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1170)    4) 
Thread[id=3660, name=qtp1080381337-3660, state=RUNNABLE, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at 
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at 
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at 
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at 
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at 
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at 
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at 
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
         at 
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) 
        at java.lang.Thread.run(Thread.java:748)    5) Thread[id=3819, 
name=coreZkRegister-894-thread-2, state=TIMED_WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
        at java.lang.Thread.run(Thread.java:748)    6) Thread[id=3814, 
name=zkCallback-998-thread-31, state=TIMED_WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
        at java.lang.Thread.run(Thread.java:748)    7) Thread[id=3816, 
name=zkCallback-998-thread-33, state=TIMED_WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
        at java.lang.Thread.run(Thread.java:748)    8) Thread[id=3774, 
name=zkCallback-998-thread-26, state=TIMED_WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
        at java.lang.Thread.run(Thread.java:748)    9) Thread[id=3762, 
name=zkCallback-998-thread-5-processing-n:127.0.0.1:42259_-EventThread, 
state=WAITING, group=TGRP-FullSolrCloudDistribCmdsTest]         at 
java.lang.Object.wait(Native Method)         at 
java.lang.Object.wait(Object.java:502)         at 
org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1406)         at 
org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1097)         at 
org.apache.solr.common.cloud.SolrZkClient.lambda$exists$3(SolrZkClient.java:308)
         at 
org.apache.solr.common.cloud.SolrZkClient$$Lambda$201/1712956663.execute(Unknown
 Source)         at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
         at 
org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:308)         
at 
org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:428)
         at org.apache.solr.cloud.ZkController$1.command(ZkController.java:328) 
        at 
org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168)
         at 
org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57)
         at 
org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142)
         at 
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)   
      at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)   
10) Thread[id=3815, name=zkCallback-998-thread-32, state=TIMED_WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
        at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 14 threads leaked from 
SUITE scope at org.apache.solr.handler.admin.PropertiesRequestHandlerTest: 
   1) Thread[id=3818, name=coreZkRegister-894-thread-1, state=TIMED_WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1406)
        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1097)
        at 
org.apache.solr.common.cloud.SolrZkClient.lambda$exists$3(SolrZkClient.java:308)
        at 
org.apache.solr.common.cloud.SolrZkClient$$Lambda$201/1712956663.execute(Unknown
 Source)
        at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at 
org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:308)
        at 
org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:93)
        at org.apache.solr.cloud.LeaderElector.setup(LeaderElector.java:367)
        at 
org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1288)
        at org.apache.solr.cloud.ZkController.register(ZkController.java:1060)
        at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:256)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$117/1778849115.run(Unknown
 Source)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=3804, name=zkCallback-998-thread-30, state=TIMED_WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1406)
        at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:781)
        at 
org.apache.solr.common.cloud.SolrZkClient.lambda$create$7(SolrZkClient.java:357)
        at 
org.apache.solr.common.cloud.SolrZkClient$$Lambda$205/1874600898.execute(Unknown
 Source)
        at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at 
org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:357)
        at org.apache.solr.cloud.Overseer.createOverseerNode(Overseer.java:765)
        at org.apache.solr.cloud.Overseer.getCollectionQueue(Overseer.java:714)
        at 
org.apache.solr.cloud.OverseerCollectionConfigSetProcessor.<init>(OverseerCollectionConfigSetProcessor.java:48)
        at org.apache.solr.cloud.Overseer.start(Overseer.java:523)
        at 
org.apache.solr.cloud.OverseerElectionContext.runLeaderProcess(ElectionContext.java:799)
        at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
        at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:56)
        at 
org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:348)
        at 
org.apache.solr.common.cloud.SolrZkClient$1.lambda$process$0(SolrZkClient.java:262)
        at 
org.apache.solr.common.cloud.SolrZkClient$1$$Lambda$191/209459966.run(Unknown 
Source)
        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$117/1778849115.run(Unknown
 Source)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=3761, 
name=zkCallback-998-thread-5-processing-n:127.0.0.1:42259_-SendThread(127.0.0.1:44052),
 state=TERMINATED, group={null group}]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:230)
        at 
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1246)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1170)
   4) Thread[id=3660, name=qtp1080381337-3660, state=RUNNABLE, 
group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at 
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
        at 
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=3819, name=coreZkRegister-894-thread-2, state=TIMED_WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=3814, name=zkCallback-998-thread-31, state=TIMED_WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   7) Thread[id=3816, name=zkCallback-998-thread-33, state=TIMED_WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   8) Thread[id=3774, name=zkCallback-998-thread-26, state=TIMED_WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   9) Thread[id=3762, 
name=zkCallback-998-thread-5-processing-n:127.0.0.1:42259_-EventThread, 
state=WAITING, group=TGRP-FullSolrCloudDistribCmdsTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1406)
        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1097)
        at 
org.apache.solr.common.cloud.SolrZkClient.lambda$exists$3(SolrZkClient.java:308)
        at 
org.apache.solr.common.cloud.SolrZkClient$$Lambda$201/1712956663.execute(Unknown
 Source)
        at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at 
org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:308)
        at 
org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:428)
        at org.apache.solr.cloud.ZkController$1.command(ZkController.java:328)
        at 
org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168)
        at 
org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57)
        at 
org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142)
        at 
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
  10) Thread[id=3815, name=zkCallback-998-thread-32, state=TIMED_WAITING, 
group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([CC3F0BBD3EEA3BFC]:0)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest

Error Message:
8 threads leaked from SUITE scope at 
org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest:     1) 
Thread[id=7707, name=StoppableIndexingThread, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
    2) Thread[id=7747, name=StoppableCommitThread-SendThread(127.0.0.1:39939), 
state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]      
   at java.lang.Thread.sleep(Native Method)         at 
org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
         at 
org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)    
     at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)    
3) Thread[id=7709, name=StoppableCommitThread, state=WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at 
java.lang.Object.wait(Native Method)         at 
java.lang.Object.wait(Object.java:502)         at 
org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1406)         at 
org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1210)         at 
org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:332)
         at 
org.apache.solr.common.cloud.SolrZkClient$$Lambda$206/828145531.execute(Unknown 
Source)         at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
         at 
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:332)        
 at 
org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1184)
         at 
org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1170)
         at 
org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:692)
         at 
org.apache.solr.client.solrj.impl.CloudSolrClient.getDocCollection(CloudSolrClient.java:1198)
         at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:840)
         at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:808)
         at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)         
at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:524)         
at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:544)         
at 
org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:49)  
  4) Thread[id=7790, name=zkCallback-1029-thread-2, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
        at java.lang.Thread.run(Thread.java:748)    5) Thread[id=7748, 
name=StoppableCommitThread-EventThread, state=WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)    
6) Thread[id=7708, name=StoppableIndexingThread, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
    7) Thread[id=7749, name=zkCallback-1029-thread-1, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
        at java.lang.Thread.run(Thread.java:748)    8) Thread[id=7791, 
name=zkCallback-1029-thread-3, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
        at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 8 threads leaked from SUITE 
scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest: 
   1) Thread[id=7707, name=StoppableIndexingThread, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   2) Thread[id=7747, name=StoppableCommitThread-SendThread(127.0.0.1:39939), 
state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at 
org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
   3) Thread[id=7709, name=StoppableCommitThread, state=WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1406)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1210)
        at 
org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:332)
        at 
org.apache.solr.common.cloud.SolrZkClient$$Lambda$206/828145531.execute(Unknown 
Source)
        at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at 
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:332)
        at 
org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1184)
        at 
org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1170)
        at 
org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:692)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.getDocCollection(CloudSolrClient.java:1198)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:840)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:808)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:524)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:544)
        at 
org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:49)
   4) Thread[id=7790, name=zkCallback-1029-thread-2, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=7748, name=StoppableCommitThread-EventThread, state=WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
   6) Thread[id=7708, name=StoppableIndexingThread, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   7) Thread[id=7749, name=zkCallback-1029-thread-1, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   8) Thread[id=7791, name=zkCallback-1029-thread-3, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([CC3F0BBD3EEA3BFC]:0)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) 
Thread[id=7707, name=StoppableIndexingThread, state=RUNNABLE, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at 
java.io.PrintStream.println(PrintStream.java:805)         at 
org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:120)
    2) Thread[id=7747, name=StoppableCommitThread-SendThread(127.0.0.1:39939), 
state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         
at java.lang.Thread.sleep(Native Method)         at 
org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
         at 
org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)    
     at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)    
3) Thread[id=7790, name=zkCallback-1029-thread-2, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
        at java.lang.Thread.run(Thread.java:748)    4) Thread[id=7708, 
name=StoppableIndexingThread, state=RUNNABLE, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at 
java.lang.Throwable.fillInStackTrace(Native Method)         at 
java.lang.Throwable.fillInStackTrace(Throwable.java:783)         at 
java.lang.Throwable.<init>(Throwable.java:265)         at 
java.lang.Exception.<init>(Exception.java:66)         at 
java.lang.RuntimeException.<init>(RuntimeException.java:62)         at 
java.lang.IllegalStateException.<init>(IllegalStateException.java:55)         
at org.apache.http.util.Asserts.check(Asserts.java:34)         at 
org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:189)         
at 
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:257)
         at 
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176)  
       at 
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)      
   at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)       
  at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) 
        at 
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
         at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
         at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
         at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:525)
         at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
         at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
         at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)         
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)       
  at 
org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)
         at 
org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)
    5) Thread[id=7749, name=zkCallback-1029-thread-1, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
        at java.lang.Thread.run(Thread.java:748)    6) Thread[id=7791, 
name=zkCallback-1029-thread-3, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
        at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie 
threads that couldn't be terminated:
   1) Thread[id=7707, name=StoppableIndexingThread, state=RUNNABLE, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.io.PrintStream.println(PrintStream.java:805)
        at 
org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:120)
   2) Thread[id=7747, name=StoppableCommitThread-SendThread(127.0.0.1:39939), 
state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at 
org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
   3) Thread[id=7790, name=zkCallback-1029-thread-2, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=7708, name=StoppableIndexingThread, state=RUNNABLE, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Throwable.fillInStackTrace(Native Method)
        at java.lang.Throwable.fillInStackTrace(Throwable.java:783)
        at java.lang.Throwable.<init>(Throwable.java:265)
        at java.lang.Exception.<init>(Exception.java:66)
        at java.lang.RuntimeException.<init>(RuntimeException.java:62)
        at java.lang.IllegalStateException.<init>(IllegalStateException.java:55)
        at org.apache.http.util.Asserts.check(Asserts.java:34)
        at 
org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:189)
        at 
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:257)
        at 
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176)
        at 
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at 
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at 
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:525)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
        at 
org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)
        at 
org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)
   5) Thread[id=7749, name=zkCallback-1029-thread-1, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=7791, name=zkCallback-1029-thread-3, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([CC3F0BBD3EEA3BFC]:0)


FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest.test

Error Message:
The Monkey ran for over 45 seconds and no jetties were stopped - this is worth 
investigating!

Stack Trace:
java.lang.AssertionError: The Monkey ran for over 45 seconds and no jetties 
were stopped - this is worth investigating!
        at 
__randomizedtesting.SeedInfo.seed([CC3F0BBD3EEA3BFC:446B346790165604]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:587)
        at 
org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest.test(ChaosMonkeySafeLeaderWithPullReplicasTest.java:174)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload

Error Message:
expected:<[{indexVersion=1509260777639,generation=2,filelist=[_5g.fdt, _5g.fdx, 
_5g.fnm, _5g.nvd, _5g.nvm, _5g.si, _5g_Asserting_0.doc, _5g_Asserting_0.tim, 
_5g_Asserting_0.tip, _5h.cfe, _5h.cfs, _5h.si, _5i.cfe, _5i.cfs, _5i.si, 
_5j.cfe, _5j.cfs, _5j.si, _5k.cfe, _5k.cfs, _5k.si, _5l.cfe, _5l.cfs, _5l.si, 
_5m.cfe, _5m.cfs, _5m.si, _5n.cfe, _5n.cfs, _5n.si, _5o.cfe, _5o.cfs, _5o.si, 
_5p.cfe, _5p.cfs, _5p.si, _5q.cfe, _5q.cfs, _5q.si, _5r.cfe, _5r.cfs, _5r.si, 
_5s.cfe, _5s.cfs, _5s.si, _5t.cfe, _5t.cfs, _5t.si, _5u.cfe, _5u.cfs, _5u.si, 
_5v.cfe, _5v.cfs, _5v.si, _5w.cfe, _5w.cfs, _5w.si, _5x.cfe, _5x.cfs, _5x.si, 
_5y.cfe, _5y.cfs, _5y.si, _5z.cfe, _5z.cfs, _5z.si, _60.cfe, _60.cfs, _60.si, 
_61.cfe, _61.cfs, _61.si, _62.cfe, _62.cfs, _62.si, _63.cfe, _63.cfs, _63.si, 
_64.cfe, _64.cfs, _64.si, _65.cfe, _65.cfs, _65.si, _66.cfe, _66.cfs, _66.si, 
_67.cfe, _67.cfs, _67.si, _69.cfe, _69.cfs, _69.si, _6a.cfe, _6a.cfs, _6a.si, 
segments_2]}]> but 
was:<[{indexVersion=1509260777639,generation=2,filelist=[_5g.fdt, _5g.fdx, 
_5g.fnm, _5g.nvd, _5g.nvm, _5g.si, _5g_Asserting_0.doc, _5g_Asserting_0.tim, 
_5g_Asserting_0.tip, _5h.cfe, _5h.cfs, _5h.si, _5i.cfe, _5i.cfs, _5i.si, 
_5j.cfe, _5j.cfs, _5j.si, _5k.cfe, _5k.cfs, _5k.si, _5l.cfe, _5l.cfs, _5l.si, 
_5m.cfe, _5m.cfs, _5m.si, _5n.cfe, _5n.cfs, _5n.si, _5o.cfe, _5o.cfs, _5o.si, 
_5p.cfe, _5p.cfs, _5p.si, _5q.cfe, _5q.cfs, _5q.si, _5r.cfe, _5r.cfs, _5r.si, 
_5s.cfe, _5s.cfs, _5s.si, _5t.cfe, _5t.cfs, _5t.si, _5u.cfe, _5u.cfs, _5u.si, 
_5v.cfe, _5v.cfs, _5v.si, _5w.cfe, _5w.cfs, _5w.si, _5x.cfe, _5x.cfs, _5x.si, 
_5y.cfe, _5y.cfs, _5y.si, _5z.cfe, _5z.cfs, _5z.si, _60.cfe, _60.cfs, _60.si, 
_61.cfe, _61.cfs, _61.si, _62.cfe, _62.cfs, _62.si, _63.cfe, _63.cfs, _63.si, 
_64.cfe, _64.cfs, _64.si, _65.cfe, _65.cfs, _65.si, _66.cfe, _66.cfs, _66.si, 
_67.cfe, _67.cfs, _67.si, _69.cfe, _69.cfs, _69.si, _6a.cfe, _6a.cfs, _6a.si, 
segments_2]}, {indexVersion=1509260777639,generation=3,filelist=[_68.fdt, 
_68.fdx, _68.fnm, _68.nvd, _68.nvm, _68.si, _68_Asserting_0.doc, 
_68_Asserting_0.tim, _68_Asserting_0.tip, _69.cfe, _69.cfs, _69.si, _6a.cfe, 
_6a.cfs, _6a.si, segments_3]}]>

Stack Trace:
java.lang.AssertionError: 
expected:<[{indexVersion=1509260777639,generation=2,filelist=[_5g.fdt, _5g.fdx, 
_5g.fnm, _5g.nvd, _5g.nvm, _5g.si, _5g_Asserting_0.doc, _5g_Asserting_0.tim, 
_5g_Asserting_0.tip, _5h.cfe, _5h.cfs, _5h.si, _5i.cfe, _5i.cfs, _5i.si, 
_5j.cfe, _5j.cfs, _5j.si, _5k.cfe, _5k.cfs, _5k.si, _5l.cfe, _5l.cfs, _5l.si, 
_5m.cfe, _5m.cfs, _5m.si, _5n.cfe, _5n.cfs, _5n.si, _5o.cfe, _5o.cfs, _5o.si, 
_5p.cfe, _5p.cfs, _5p.si, _5q.cfe, _5q.cfs, _5q.si, _5r.cfe, _5r.cfs, _5r.si, 
_5s.cfe, _5s.cfs, _5s.si, _5t.cfe, _5t.cfs, _5t.si, _5u.cfe, _5u.cfs, _5u.si, 
_5v.cfe, _5v.cfs, _5v.si, _5w.cfe, _5w.cfs, _5w.si, _5x.cfe, _5x.cfs, _5x.si, 
_5y.cfe, _5y.cfs, _5y.si, _5z.cfe, _5z.cfs, _5z.si, _60.cfe, _60.cfs, _60.si, 
_61.cfe, _61.cfs, _61.si, _62.cfe, _62.cfs, _62.si, _63.cfe, _63.cfs, _63.si, 
_64.cfe, _64.cfs, _64.si, _65.cfe, _65.cfs, _65.si, _66.cfe, _66.cfs, _66.si, 
_67.cfe, _67.cfs, _67.si, _69.cfe, _69.cfs, _69.si, _6a.cfe, _6a.cfs, _6a.si, 
segments_2]}]> but 
was:<[{indexVersion=1509260777639,generation=2,filelist=[_5g.fdt, _5g.fdx, 
_5g.fnm, _5g.nvd, _5g.nvm, _5g.si, _5g_Asserting_0.doc, _5g_Asserting_0.tim, 
_5g_Asserting_0.tip, _5h.cfe, _5h.cfs, _5h.si, _5i.cfe, _5i.cfs, _5i.si, 
_5j.cfe, _5j.cfs, _5j.si, _5k.cfe, _5k.cfs, _5k.si, _5l.cfe, _5l.cfs, _5l.si, 
_5m.cfe, _5m.cfs, _5m.si, _5n.cfe, _5n.cfs, _5n.si, _5o.cfe, _5o.cfs, _5o.si, 
_5p.cfe, _5p.cfs, _5p.si, _5q.cfe, _5q.cfs, _5q.si, _5r.cfe, _5r.cfs, _5r.si, 
_5s.cfe, _5s.cfs, _5s.si, _5t.cfe, _5t.cfs, _5t.si, _5u.cfe, _5u.cfs, _5u.si, 
_5v.cfe, _5v.cfs, _5v.si, _5w.cfe, _5w.cfs, _5w.si, _5x.cfe, _5x.cfs, _5x.si, 
_5y.cfe, _5y.cfs, _5y.si, _5z.cfe, _5z.cfs, _5z.si, _60.cfe, _60.cfs, _60.si, 
_61.cfe, _61.cfs, _61.si, _62.cfe, _62.cfs, _62.si, _63.cfe, _63.cfs, _63.si, 
_64.cfe, _64.cfs, _64.si, _65.cfe, _65.cfs, _65.si, _66.cfe, _66.cfs, _66.si, 
_67.cfe, _67.cfs, _67.si, _69.cfe, _69.cfs, _69.si, _6a.cfe, _6a.cfs, _6a.si, 
segments_2]}, {indexVersion=1509260777639,generation=3,filelist=[_68.fdt, 
_68.fdx, _68.fnm, _68.nvd, _68.nvm, _68.si, _68_Asserting_0.doc, 
_68_Asserting_0.tim, _68_Asserting_0.tip, _69.cfe, _69.cfs, _69.si, _6a.cfe, 
_6a.cfs, _6a.si, segments_3]}]>
        at 
__randomizedtesting.SeedInfo.seed([CC3F0BBD3EEA3BFC:E9E8108D4EA235FF]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:147)
        at 
org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload(TestReplicationHandler.java:1277)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.search.function.TestFunctionQuery.testLongComparisons

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
        at 
__randomizedtesting.SeedInfo.seed([CC3F0BBD3EEA3BFC:CFA794DFF08C6C0E]:0)
        at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:885)
        at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:845)
        at 
org.apache.solr.search.function.TestFunctionQuery.singleTest(TestFunctionQuery.java:131)
        at 
org.apache.solr.search.function.TestFunctionQuery.singleTest(TestFunctionQuery.java:137)
        at 
org.apache.solr.search.function.TestFunctionQuery.testLongComparisons(TestFunctionQuery.java:943)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: REQUEST FAILED: 
xpath=//doc[./str[@name='id']='2'  and ./float[@name='score']='2.0']
        xml response was: <?xml version="1.0" encoding="UTF-8"?>
<response>

<lst name="responseHeader">
  <int name="status">0</int>
  <int name="QTime">0</int>
</lst>
<result name="response" numFound="112" start="0" maxScore="5.0">
  <doc>
    <str name="id">1</str>
    <int name="id_i">1</int>
    <long name="number_of_atoms_in_universe_l">9223372036854775807</long>
    <float name="score">5.0</float></doc>
  <doc>
    <str name="id">0</str>
    <int name="id_i">0</int>
    <float name="default_amount_f">13029.0</float>
    <str name="amounts_dpf">1|1.14 2|2.16 3|3.60 4|4.26 5|5.72 6|6.84 7|7.50 
8|8.90 9|9.86 10|10.96 11|11.51 12|12.31 13|13.90 14|14.48 15|15.0 16|16.81 
17|17.4 18|18.4 19|19.17 20|20.89 21|21.49 22|22.4 23|23.63 24|24.96 25|25.26 
26|26.95 27|27.35 28|28.40 29|29.47 30|30.10 31|31.90 32|32.11 33|33.52 
34|34.37 35|35.63 36|36.27 37|37.15 38|38.38 39|39.2 40|40.47 41|41.40 42|42.11 
43|43.85 44|44.43 45|45.36 46|46.45 47|47.19 48|48.1 49|49.36 50|50.76 51|51.74 
52|52.77 53|53.88 54|54.99 55|55.25 56|56.43 57|57.22 58|58.53 59|59.29 
60|60.17 61|61.58 62|62.48 63|63.87 64|64.3 65|65.24 66|66.12 67|67.19 68|68.65 
69|69.55 70|70.59 71|71.53 72|72.95 73|73.79 74|74.70 75|75.47 76|76.1 77|77.6 
78|78.33 79|79.37 80|80.11 81|81.72 82|82.6 83|83.83 84|84.24 85|85.38 86|86.6 
87|87.48 88|88.32 89|89.49 90|90.17 91|91.69 92|92.85 93|93.28 94|94.9 95|95.83 
96|96.20 97|97.54 98|98.8 99|99.86 100|100.82 101|101.27 102|102.67 103|103.16 
104|104.68 105|105.95 106|106.75 107|107.0 108|108.54 109|109.10 110|110.55 
111|111.14 112|112.14 113|113.85 114|114.43 115|115.13 116|116.45 117|117.72 
118|118.55 119|119.55 120|120.35 121|121.28 122|122.19 123|123.53 124|124.9 
125|125.97 126|126.16 127|127.98 128|128.47 129|129.52 130|130.26 131|131.19 
132|132.44 133|133.35 134|134.78 135|135.64 136|136.33 137|137.69 138|138.85 
139|139.23 140|140.25 141|141.17 142|142.77 143|143.65 144|144.94 145|145.89 
146|146.57 147|147.31 148|148.94 149|149.33 150|150.17 151|151.78 152|152.34 
153|153.65 154|154.96 155|155.18 156|156.93 157|157.55 158|158.9 159|159.93 
160|160.88 161|161.92 162|162.34 163|163.59 164|164.10 165|165.95 166|166.7 
167|167.54 168|168.16 169|169.8 170|170.42 171|171.35 172|172.88 173|173.20 
174|174.88 175|175.77 176|176.90 177|177.83 178|178.90 179|179.52 180|180.38 
181|181.10 182|182.50 183|183.39 184|184.9 185|185.81 186|186.87 187|187.65 
188|188.5 189|189.33 190|190.8 191|191.86 192|192.21 193|193.67 194|194.55 
195|195.94 196|196.11 197|197.27 198|198.24 199|199.77 200|200.31 201|201.77 
202|202.80 203|203.53 204|204.6 205|205.76 206|206.64 207|207.14 208|208.65 
209|209.6 210|210.69 211|211.86 212|212.16 213|213.62 214|214.0 215|215.81 
216|216.49 217|217.40 218|218.62 219|219.71 220|220.17 221|221.27 222|222.15 
223|223.14 224|224.30 225|225.57 226|226.43 227|227.69 228|228.39 229|229.18 
230|230.96 231|231.23 232|232.27 233|233.49 234|234.61 235|235.85 236|236.93 
237|237.67 238|238.6 239|239.53 240|240.26 241|241.9 242|242.35 243|243.91 
244|244.17 245|245.14 246|246.84 247|247.5 248|248.54 249|249.46 250|250.86 
251|251.84 252|252.93 253|253.66 254|254.13 255|255.54 256|256.65 257|257.46 
258|258.65 259|259.77 260|260.99 261|261.25 262|262.86 263|263.75 264|264.13 
265|265.17 266|266.47 267|267.97 268|268.84 269|269.17 270|270.71 271|271.66 
272|272.67 273|273.31 274|274.86 275|275.7 276|276.32 277|277.38 278|278.2 
279|279.21 280|280.54 281|281.18 282|282.63 283|283.48 284|284.60 285|285.70 
286|286.48 287|287.39 288|288.95 289|289.10 290|290.97 291|291.27 292|292.93 
293|293.74 294|294.37 295|295.46 296|296.40 297|297.5 298|298.32 299|299.56 
300|300.42 301|301.23 302|302.0 303|303.80 304|304.48 305|305.93 306|306.58 
307|307.45 308|308.83 309|309.90 310|310.51 311|311.79 312|312.39 313|313.83 
314|314.3 315|315.43 316|316.64 317|317.82 318|318.50 319|319.96 320|320.98 
321|321.77 322|322.11 323|323.88 324|324.88 325|325.81 326|326.3 327|327.13 
328|328.12 329|329.8 330|330.26 331|331.44 332|332.23 333|333.86 334|334.89 
335|335.99 336|336.3 337|337.88 338|338.24 339|339.67 340|340.63 341|341.31 
342|342.21 343|343.43 344|344.16 345|345.18 346|346.96 347|347.12 348|348.22 
349|349.99 350|350.6 351|351.6 352|352.72 353|353.18 354|354.54 355|355.29 
356|356.62 357|357.13 358|358.64 359|359.36 360|360.81 361|361.3 362|362.12 
363|363.28 364|364.25 365|365.79 366|366.76 367|367.38 368|368.93 369|369.46 
370|370.68 371|371.34 372|372.12 373|373.5 374|374.13 375|375.77 376|376.48 
377|377.64 378|378.61 379|379.42 380|380.68 381|381.27 382|382.57 383|383.79 
384|384.78 385|385.60 386|386.67 387|387.88 388|388.15 389|389.15 390|390.83 
391|391.14 392|392.56 393|393.30 394|394.86 395|395.97 396|396.58 397|397.63 
398|398.36 399|399.60 400|400.47 401|401.70 402|402.42 403|403.65 404|404.74 
405|405.95 406|406.64 407|407.81 408|408.62 409|409.0 410|410.99 411|411.25 
412|412.80 413|413.95 414|414.19 415|415.18 416|416.20 417|417.74 418|418.28 
419|419.91 420|420.94 421|421.62 422|422.39 423|423.70 424|424.8 425|425.33 
426|426.66 427|427.5 428|428.23 429|429.14 430|430.39 431|431.75 432|432.77 
433|433.51 434|434.52 435|435.94 436|436.81 437|437.84 438|438.4 439|439.55 
440|440.93 441|441.80 442|442.38 443|443.63 444|444.94 445|445.57 446|446.1 
447|447.0 448|448.17 449|449.2 450|450.85 451|451.22 452|452.63 453|453.83 
454|454.78 455|455.11 456|456.46 457|457.82 458|458.51 459|459.85 460|460.75 
461|461.74 462|462.66 463|463.35 464|464.20 465|465.41 466|466.56 467|467.53 
468|468.12 469|469.82 470|470.64 471|471.19 472|472.12 473|473.46 474|474.96 
475|475.10 476|476.20 477|477.91 478|478.64 479|479.71 480|480.68 481|481.0 
482|482.37 483|483.35 484|484.32 485|485.15 486|486.11 487|487.15 488|488.87 
489|489.75 490|490.55 491|491.16 492|492.26 493|493.67 494|494.29 495|495.21 
496|496.63 497|497.75 498|498.63 499|499.90 500|500.4 501|501.68 502|502.20 
503|503.89 504|504.33 505|505.17 506|506.53 507|507.53 508|508.55 509|509.47 
510|510.4 511|511.46 512|512.98 513|513.30 514|514.97 515|515.51 516|516.73 
517|517.70 518|518.84 519|519.64 520|520.75 521|521.30 522|522.85 523|523.98 
524|524.37 525|525.75 526|526.77 527|527.82 528|528.58 529|529.80 530|530.1 
531|531.88 532|532.20 533|533.35 534|534.75 535|535.44 536|536.82 537|537.79 
538|538.20 539|539.20 540|540.87 541|541.44 542|542.99 543|543.67 544|544.54 
545|545.28 546|546.97 547|547.13 548|548.45 549|549.39 550|550.13 551|551.10 
552|552.0 553|553.21 554|554.84 555|555.69 556|556.17 557|557.70 558|558.16 
559|559.98 560|560.94 561|561.68 562|562.47 563|563.67 564|564.84 565|565.85 
566|566.83 567|567.6 568|568.59 569|569.92 570|570.61 571|571.11 572|572.35 
573|573.61 574|574.17 575|575.63 576|576.6 577|577.7 578|578.98 579|579.9 
580|580.73 581|581.52 582|582.0 583|583.76 584|584.10 585|585.60 586|586.87 
587|587.73 588|588.85 589|589.27 590|590.90 591|591.61 592|592.29 593|593.48 
594|594.31 595|595.26 596|596.25 597|597.85 598|598.64 599|599.63 600|600.41 
601|601.22 602|602.79 603|603.27 604|604.88 605|605.27 606|606.32 607|607.68 
608|608.37 609|609.26 610|610.94 611|611.14 612|612.20 613|613.16 614|614.99 
615|615.63 616|616.6 617|617.20 618|618.89 619|619.2 620|620.70 621|621.21 
622|622.40 623|623.26 624|624.89 625|625.73 626|626.71 627|627.71 628|628.62 
629|629.85 630|630.74 631|631.72 632|632.78 633|633.12 634|634.67 635|635.33 
636|636.62 637|637.33 638|638.95 639|639.17 640|640.14 641|641.40 642|642.89 
643|643.41 644|644.53 645|645.55 646|646.52 647|647.52 648|648.47 649|649.4 
650|650.16 651|651.30 652|652.23 653|653.97 654|654.48 655|655.43 656|656.23 
657|657.66 658|658.25 659|659.32 660|660.46 661|661.43 662|662.0 663|663.7 
664|664.92 665|665.5 666|666.91 667|667.27 668|668.96 669|669.53 670|670.85 
671|671.5 672|672.4 673|673.37 674|674.81 675|675.34 676|676.75 677|677.0 
678|678.28 679|679.65 680|680.32 681|681.0 682|682.79 683|683.29 684|684.75 
685|685.30 686|686.57 687|687.78 688|688.41 689|689.90 690|690.45 691|691.23 
692|692.21 693|693.68 694|694.80 695|695.33 696|696.12 697|697.82 698|698.43 
699|699.46 700|700.8 701|701.94 702|702.7 703|703.1 704|704.9 705|705.69 
706|706.25 707|707.39 708|708.58 709|709.67 710|710.62 711|711.10 712|712.40 
713|713.67 714|714.99 715|715.47 716|716.2 717|717.2 718|718.29 719|719.63 
720|720.92 721|721.64 722|722.44 723|723.1 724|724.65 725|725.41 726|726.80 
727|727.98 728|728.64 729|729.19 730|730.32 731|731.81 732|732.95 733|733.2 
734|734.40 735|735.39 736|736.11 737|737.10 738|738.87 739|739.30 740|740.17 
741|741.25 742|742.55 743|743.1 744|744.61 745|745.57 746|746.2 747|747.9 
748|748.34 749|749.36 750|750.68 751|751.77 752|752.84 753|753.68 754|754.39 
755|755.64 756|756.19 757|757.8 758|758.72 759|759.92 760|760.83 761|761.94 
762|762.14 763|763.98 764|764.80 765|765.38 766|766.14 767|767.26 768|768.71 
769|769.18 770|770.21 771|771.9 772|772.22 773|773.74 774|774.96 775|775.2 
776|776.34 777|777.79 778|778.14 779|779.7 780|780.25 781|781.95 782|782.62 
783|783.36 784|784.14 785|785.75 786|786.76 787|787.5 788|788.56 789|789.26 
790|790.19 791|791.77 792|792.30 793|793.77 794|794.93 795|795.83 796|796.13 
797|797.62 798|798.92 799|799.58 800|800.43 801|801.94 802|802.70 803|803.99 
804|804.47 805|805.14 806|806.37 807|807.80 808|808.60 809|809.52 810|810.2 
811|811.1 812|812.0 813|813.32 814|814.58 815|815.25 816|816.25 817|817.14 
818|818.0 819|819.19 820|820.45 821|821.99 822|822.99 823|823.3 824|824.84 
825|825.72 826|826.17 827|827.60 828|828.87 829|829.2 830|830.31 831|831.53 
832|832.57 833|833.47 834|834.29 835|835.73 836|836.39 837|837.33 838|838.1 
839|839.4 840|840.94 841|841.71 842|842.0 843|843.59 844|844.59 845|845.23 
846|846.20 847|847.89 848|848.3 849|849.8 850|850.58 851|851.24 852|852.51 
853|853.15 854|854.1 855|855.41 856|856.41 857|857.14 858|858.51 859|859.54 
860|860.74 861|861.47 862|862.46 863|863.29 864|864.10 865|865.68 866|866.2 
867|867.22 868|868.69 869|869.77 870|870.42 871|871.66 872|872.11 873|873.75 
874|874.78 875|875.58 876|876.37 877|877.81 878|878.23 879|879.45 880|880.18 
881|881.86 882|882.89 883|883.55 884|884.96 885|885.41 886|886.92 887|887.14 
888|888.75 889|889.94 890|890.87 891|891.55 892|892.30 893|893.52 894|894.99 
895|895.52 896|896.34 897|897.76 898|898.27 899|899.35 900|900.68 901|901.70 
902|902.26 903|903.10 904|904.39 905|905.91 906|906.66 907|907.35 908|908.23 
909|909.75 910|910.13 911|911.53 912|912.44 913|913.78 914|914.4 915|915.3 
916|916.79 917|917.54 918|918.93 919|919.34 920|920.18 921|921.55 922|922.13 
923|923.78 924|924.73 925|925.8 926|926.26 927|927.92 928|928.9 929|929.30 
930|930.28 931|931.97 932|932.92 933|933.70 934|934.17 935|935.89 936|936.39 
937|937.90 938|938.40 939|939.24 940|940.97 941|941.54 942|942.49 943|943.21 
944|944.16 945|945.5 946|946.94 947|947.71 948|948.13 949|949.28 950|950.69 
951|951.69 952|952.10 953|953.92 954|954.22 955|955.7 956|956.93 957|957.32 
958|958.9 959|959.16 960|960.2 961|961.20 962|962.7 963|963.47 964|964.82 
965|965.36 966|966.59 967|967.74 968|968.23 969|969.74 970|970.88 971|971.8 
972|972.41 973|973.68 974|974.9 975|975.71 976|976.50 977|977.89 978|978.71 
979|979.74 980|980.12 981|981.62 982|982.24 983|983.10 984|984.18 985|985.58 
986|986.69 987|987.16 988|988.88 989|989.19 990|990.99 991|991.66 992|992.52 
993|993.29 994|994.40 995|995.60 996|996.70 997|997.25 998|998.69 999|999.52 
1000|1000.58 1001|1001.80 1002|1002.70 1003|1003.64 1004|1004.29 1005|1005.55 
1006|1006.0 1007|1007.47 1008|1008.70 1009|1009.96 1010|1010.61 1011|1011.98 
1012|1012.78 1013|1013.27 1014|1014.93 1015|1015.73 1016|1016.90 1017|1017.62 
1018|1018.89 1019|1019.93 1020|1020.94 1021|1021.68 1022|1022.96 1023|1023.2 
1024|1024.20 1025|1025.67 1026|1026.40 1027|1027.28 1028|1028.67 1029|1029.58 
1030|1030.13 1031|1031.27 1032|1032.58 1033|1033.95 1034|1034.76 1035|1035.95 
1036|1036.53 1037|1037.14 1038|1038.38 1039|1039.87 1040|1040.32 1041|1041.29 
1042|1042.84 1043|1043.89 1044|1044.18 1045|1045.7 1046|1046.41 1047|1047.83 
1048|1048.31 1049|1049.60 1050|1050.38 1051|1051.90 1052|1052.38 1053|1053.16 
1054|1054.35 1055|1055.47 1056|1056.9 1057|1057.88 1058|1058.19 1059|1059.42 
1060|1060.36 1061|1061.50 1062|1062.19 1063|1063.38 1064|1064.70 1065|1065.92 
1066|1066.24 1067|1067.45 1068|1068.70 1069|1069.84 1070|1070.83 1071|1071.29 
1072|1072.22 1073|1073.84 1074|1074.95 1075|1075.92 1076|1076.19 1077|1077.62 
1078|1078.98 1079|1079.65 1080|1080.82 1081|1081.94 1082|1082.36 1083|1083.79 
1084|1084.26 1085|1085.60 1086|1086.41 1087|1087.35 1088|1088.57 1089|1089.88 
1090|1090.24 1091|1091.87 1092|1092.68 1093|1093.72 1094|1094.10 1095|1095.94 
1096|1096.85 1097|1097.68 1098|1098.83 1099|1099.89 1100|1100.41 1101|1101.42 
1102|1102.67 1103|1103.25 1104|1104.70 1105|1105.8 1106|1106.38 1107|1107.87 
1108|1108.35 1109|1109.14 1110|1110.75 1111|1111.99 1112|1112.41 1113|1113.31 
1114|1114.63 1115|1115.36 1116|1116.91 1117|1117.50 1118|1118.15 1119|1119.32 
1120|1120.97 1121|1121.93 1122|1122.74 1123|1123.1 1124|1124.11 1125|1125.84 
1126|1126.88 1127|1127.53 1128|1128.43 1129|1129.40 1130|1130.79 1131|1131.83 
1132|1132.90 1133|1133.27 1134|1134.60 1135|1135.44 1136|1136.8 1137|1137.79 
1138|1138.30 1139|1139.52 1140|1140.66 1141|1141.60 1142|1142.59 1143|1143.55 
1144|1144.5 1145|1145.38 1146|1146.77 1147|1147.39 1148|1148.74 1149|1149.57 
1150|1150.73 1151|1151.27 1152|1152.45 1153|1153.9 1154|1154.45 1155|1155.31 
1156|1156.72 1157|1157.88 1158|1158.95 1159|1159.87 1160|1160.86 1161|1161.79 
1162|1162.30 1163|1163.10 1164|1164.68 1165|1165.32 1166|1166.95 1167|1167.36 
1168|1168.25 1169|1169.7 1170|1170.78 1171|1171.42 1172|1172.18 1173|1173.67 
1174|1174.25 1175|1175.5 1176|1176.56 1177|1177.41 1178|1178.9 1179|1179.82 
1180|1180.55 1181|1181.93 1182|1182.48 1183|1183.42 1184|1184.82 1185|1185.63 
1186|1186.55 1187|1187.19 1188|1188.74 1189|1189.7 1190|1190.22 1191|1191.65 
1192|1192.94 1193|1193.37 1194|1194.29 1195|1195.38 1196|1196.41 1197|1197.3 
1198|1198.37 1199|1199.79 1200|1200.19 1201|1201.77 1202|1202.29 1203|1203.98 
1204|1204.42 1205|1205.41 1206|1206.89 1207|1207.70 1208|1208.34 1209|1209.45 
1210|1210.99 1211|1211.71 1212|1212.31 1213|1213.94 1214|1214.14 1215|1215.67 
1216|1216.68 1217|1217.59 1218|1218.93 1219|1219.85 1220|1220.94 1221|1221.60 
1222|1222.59 1223|1223.36 1224|1224.5 1225|1225.11 1226|1226.87 1227|1227.31 
1228|1228.74 1229|1229.17 1230|1230.75 1231|1231.6 1232|1232.38 1233|1233.36 
1234|1234.60 1235|1235.69 1236|1236.92 1237|1237.99 1238|1238.70 1239|1239.32 
1240|1240.82 1241|1241.74 1242|1242.52 1243|1243.81 1244|1244.16 1245|1245.69 
1246|1246.52 1247|1247.95 1248|1248.42 1249|1249.8 1250|1250.16 1251|1251.88 
1252|1252.61 1253|1253.93 1254|1254.22 1255|1255.35 1256|1256.77 1257|1257.0 
1258|1258.62 1259|1259.97 1260|1260.16 1261|1261.20 1262|1262.90 1263|1263.59 
1264|1264.99 1265|1265.1 1266|1266.88 1267|1267.66 1268|1268.6 1269|1269.0 
1270|1270.22 1271|1271.31 1272|1272.17 1273|1273.74 1274|1274.3 1275|1275.12 
1276|1276.64 1277|1277.90 1278|1278.45 1279|1279.4 1280|1280.60 1281|1281.26 
1282|1282.92 1283|1283.52 1284|1284.22 1285|1285.43 1286|1286.66 1287|1287.3 
1288|1288.77 1289|1289.34 1290|1290.43 1291|1291.17 1292|1292.71 1293|1293.1 
1294|1294.91 1295|1295.80 1296|1296.58 1297|1297.54 1298|1298.91 1299|1299.47 
1300|1300.5 1301|1301.26 1302|1302.85 1303|1303.3 1304|1304.38 1305|1305.54 
1306|1306.53 1307|1307.95 1308|1308.58 1309|1309.68 1310|1310.28 1311|1311.32 
1312|1312.19 1313|1313.60 1314|1314.42 1315|1315.17 1316|1316.33 1317|1317.15 
1318|1318.66 1319|1319.91 1320|1320.57 1321|1321.24 1322|1322.59 1323|1323.58 
1324|1324.64 1325|1325.62 1326|1326.70 1327|1327.53 1328|1328.95 1329|1329.75 
1330|1330.7 1331|1331.78 1332|1332.54 1333|1333.80 1334|1334.41 1335|1335.46 
1336|1336.87 1337|1337.31 1338|1338.10 1339|1339.66 1340|1340.0 1341|1341.11 
1342|1342.39 1343|1343.21 1344|1344.76 1345|1345.4 1346|1346.23 1347|1347.91 
1348|1348.14 1349|1349.16 1350|1350.92 1351|1351.4 1352|1352.45 1353|1353.96 
1354|1354.98 1355|1355.15 1356|1356.72 1357|1357.84 1358|1358.2 1359|1359.38 
1360|1360.83 1361|1361.92 1362|1362.96 1363|1363.94 1364|1364.0 1365|1365.33 
1366|1366.0 1367|1367.51 1368|1368.15 1369|1369.50 1370|1370.81 1371|1371.28 
1372|1372.14 1373|1373.21 1374|1374.9 1375|1375.9 1376|1376.60 1377|1377.60 
1378|1378.71 1379|1379.81 1380|1380.83 1381|1381.55 1382|1382.53 1383|1383.7 
1384|1384.89 1385|1385.35 1386|1386.22 1387|1387.14 1388|1388.2 1389|1389.20 
1390|1390.5 1391|1391.11 1392|1392.81 1393|1393.15 1394|1394.9 1395|1395.56 
1396|1396.94 1397|1397.86 1398|1398.62 1399|1399.18 1400|1400.60 1401|1401.49 
1402|1402.95 1403|1403.20 1404|1404.15 1405|1405.27 1406|1406.2 1407|1407.33 
1408|1408.96 1409|1409.88 1410|1410.43 1411|1411.4 1412|1412.13 1413|1413.28 
1414|1414.81 1415|1415.51 1416|1416.45 1417|1417.73 1418|1418.63 1419|1419.13 
1420|1420.11 1421|1421.39 1422|1422.18 1423|1423.86 1424|1424.84 1425|1425.89 
1426|1426.15 1427|1427.45 1428|1428.3 1429|1429.42 1430|1430.38 1431|1431.43 
1432|1432.86 1433|1433.80 1434|1434.33 1435|1435.37 1436|1436.63 1437|1437.26 
1438|1438.61 1439|1439.0 1440|1440.67 1441|1441.91 1442|1442.91 1443|1443.57 
1444|1444.82 1445|1445.33 1446|1446.49 1447|1447.1 1448|1448.8 1449|1449.19 
1450|1450.45 1451|1451.77 1452|1452.72 1453|1453.72 1454|1454.67 1455|1455.64 
1456|1456.7 1457|1457.44 1458|1458.33 1459|1459.45 1460|1460.60 1461|1461.34 
1462|1462.98 1463|1463.94 1464|1464.87 1465|1465.37 1466|1466.43 1467|1467.48 
1468|1468.52 1469|1469.36 1470|1470.71 1471|1471.75 1472|1472.79 1473|1473.98 
1474|1474.14 1475|1475.64 1476|1476.64 1477|1477.91 1478|1478.36 1479|1479.1 
1480|1480.9 1481|1481.29 1482|1482.2 1483|1483.74 1484|1484.7 1485|1485.99 
1486|1486.83 1487|1487.31 1488|1488.30 1489|1489.29 1490|1490.86 1491|1491.22 
1492|1492.64 1493|1493.47 1494|1494.74 1495|1495.37 1496|1496.32 1497|1497.80 
1498|1498.7 1499|1499.49 1500|1500.23 1501|1501.13 1502|1502.6 1503|1503.52 
1504|1504.58 1505|1505.8 1506|1506.47 1507|1507.43 1508|1508.20 1509|1509.43 
1510|1510.21 1511|1511.15 1512|1512.14 1513|1513.18 1514|1514.89 1515|1515.51 
1516|1516.96 1517|1517.44 1518|1518.13 1519|1519.22 1520|1520.76 1521|1521.61 
1522|1522.40 1523|1523.51 1524|1524.49 1525|1525.53 1526|1526.51 1527|1527.5 
1528|1528.37 1529|1529.51 1530|1530.74 1531|1531.49 1532|1532.87 1533|1533.79 
1534|1534.62 1535|1535.90 1536|1536.57 1537|1537.58 1538|1538.6 1539|1539.32 
1540|1540.24 1541|1541.70 1542|1542.99 1543|1543.96 1544|1544.97 1545|1545.77 
1546|1546.96 1547|1547.46 1548|1548.92 1549|1549.42 1550|1550.68 1551|1551.79 
1552|1552.41 1553|1553.1 1554|1554.61 1555|1555.69 1556|1556.7 1557|1557.4 
1558|1558.98 1559|1559.86 1560|1560.44 1561|1561.95 1562|1562.43 1563|1563.59 
1564|1564.54 1565|1565.16 1566|1566.60 1567|1567.53 1568|1568.63 1569|1569.41 
1570|1570.16 1571|1571.75 1572|1572.75 1573|1573.93 1574|1574.14 1575|1575.38 
1576|1576.53 1577|1577.64 1578|1578.65 1579|1579.41 1580|1580.50 1581|1581.97 
1582|1582.73 1583|1583.45 1584|1584.41 1585|1585.84 1586|1586.80 1587|1587.37 
1588|1588.26 1589|1589.42 1590|1590.67 1591|1591.92 1592|1592.72 1593|1593.21 
1594|1594.21 1595|1595.29 1596|1596.26 1597|1597.63 1598|1598.18 1599|1599.69 
1600|1600.62 1601|1601.97 1602|1602.78 1603|1603.43 1604|1604.53 1605|1605.42 
1606|1606.79 1607|1607.29 1608|1608.6 1609|1609.99 1610|1610.17 1611|1611.0 
1612|1612.43 1613|1613.51 1614|1614.52 1615|1615.8 1616|1616.42 1617|1617.98 
1618|1618.58 1619|1619.24 1620|1620.33 1621|1621.60 1622|1622.82 1623|1623.97 
1624|1624.19 1625|1625.84 1626|1626.13 1627|1627.41 1628|1628.2 1629|1629.91 
1630|1630.69 1631|1631.48 1632|1632.15 1633|1633.47 1634|1634.60 1635|1635.29 
1636|1636.12 1637|1637.16 1638|1638.91 1639|1639.9 1640|1640.84 1641|1641.70 
1642|1642.48 1643|1643.29 1644|1644.18 1645|1645.20 1646|1646.45 1647|1647.8 
1648|1648.82 1649|1649.1 1650|1650.64 1651|1651.20 1652|1652.14 1653|1653.68 
1654|1654.65 1655|1655.9 1656|1656.72 1657|1657.49 1658|1658.87 1659|1659.2 
1660|1660.83 1661|1661.29 1662|1662.59 1663|1663.23 1664|1664.81 1665|1665.73 
1666|1666.88 1667|1667.23 1668|1668.64 1669|1669.26 1670|1670.80 1671|1671.58 
1672|1672.28 1673|1673.19 1674|1674.32 1675|1675.8 1676|1676.26 1677|1677.30 
1678|1678.89 1679|1679.10 1680|1680.39 1681|1681.71 1682|1682.0 1683|1683.36 
1684|1684.96 1685|1685.26 1686|1686.99 1687|1687.26 1688|1688.6 1689|1689.43 
1690|1690.78 1691|1691.28 1692|1692.34 1693|1693.50 1694|1694.87 1695|1695.8 
1696|1696.14 1697|1697.3 1698|1698.18 1699|1699.87 1700|1700.32 1701|1701.22 
1702|1702.39 1703|1703.13 1704|1704.33 1705|1705.43 1706|1706.4 1707|1707.21 
1708|1708.68 1709|1709.34 1710|1710.30 1711|1711.11 1712|1712.54 1713|1713.27 
1714|1714.20 1715|1715.87 1716|1716.81 1717|1717.28 1718|1718.57 1719|1719.83 
1720|1720.62 1721|1721.68 1722|1722.3 1723|1723.57 1724|1724.98 1725|1725.5 
1726|1726.9 1727|1727.27 1728|1728.41 1729|1729.34 1730|1730.45 1731|1731.12 
1732|1732.89 1733|1733.30 1734|1734.71 1735|1735.8 1736|1736.23 1737|1737.28 
1738|1738.16 1739|1739.70 1740|1740.23 1741|1741.82 1742|1742.67 1743|1743.19 
1744|1744.86 1745|1745.21 1746|1746.55 1747|1747.63 1748|1748.20 1749|1749.21 
1750|1750.48 1751|1751.47 1752|1752.96 1753|1753.57 1754|1754.97 1755|1755.74 
1756|1756.86 1757|1757.35 1758|1758.2 1759|1759.60 1760|1760.71 1761|1761.18 
1762|1762.52 1763|1763.55 1764|1764.0 1765|1765.68 1766|1766.19 1767|1767.62 
1768|1768.41 1769|1769.30 1770|1770.46 1771|1771.70 1772|1772.83 1773|1773.68 
1774|1774.98 1775|1775.87 1776|1776.2 1777|1777.87 1778|1778.67 1779|1779.73 
1780|1780.35 1781|1781.66 1782|1782.17 1783|1783.96 1784|1784.76 1785|1785.8 
1786|1786.71 1787|1787.57 1788|1788.83 1789|1789.14 1790|1790.80 1791|1791.44 
1792|1792.65 1793|1793.64 1794|1794.20 1795|1795.96 1796|1796.51 1797|1797.88 
1798|1798.49 1799|1799.75 1800|1800.37 1801|1801.81 1802|1802.14 1803|1803.3 
1804|1804.49 1805|1805.61 1806|1806.42 1807|1807.15 1808|1808.52 1809|1809.84 
1810|1810.21 1811|1811.36 1812|1812.82 1813|1813.32 1814|1814.86 1815|1815.35 
1816|1816.18 1817|1817.57 1818|1818.74 1819|1819.8 1820|1820.60 1821|1821.65 
1822|1822.72 1823|1823.17 1824|1824.60 1825|1825.59 1826|1826.50 1827|1827.23 
1828|1828.69 1829|1829.47 1830|1830.45 1831|1831.12 1832|1832.15 1833|1833.89 
1834|1

[...truncated too long message...]

bleDirectoryReader(UninvertingDirectoryReader(Uninverting(_1c(7.2.0):C25/2:delGen=1)
 Uninverting(_1d(7.2.0):c3)))}
   [junit4]   2> 11100263 INFO  
(TEST-TestFunctionQuery.testDegreeRads-seed#[CC3F0BBD3EEA3BFC]) [    ] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null 
params={}{commit=} 0 50
   [junit4]   2> 11100264 INFO  
(TEST-TestFunctionQuery.testDegreeRads-seed#[CC3F0BBD3EEA3BFC]) [    ] 
o.a.s.c.S.Request [collection1]  webapp=null path=null 
params={q={!func}rad(x_td)&fl=*,score&fq=id:1&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 11100264 INFO  
(TEST-TestFunctionQuery.testDegreeRads-seed#[CC3F0BBD3EEA3BFC]) [    ] 
o.a.s.c.S.Request [collection1]  webapp=null path=null 
params={q={!func}rad(x_td)&fl=*,score&fq=id:2&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 11100265 INFO  
(TEST-TestFunctionQuery.testDegreeRads-seed#[CC3F0BBD3EEA3BFC]) [    ] 
o.a.s.c.S.Request [collection1]  webapp=null path=null 
params={q={!func}rad(x_td)&fl=*,score&fq=id:3&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 11100265 INFO  
(TEST-TestFunctionQuery.testDegreeRads-seed#[CC3F0BBD3EEA3BFC]) [    ] 
o.a.s.c.S.Request [collection1]  webapp=null path=null 
params={q={!func}deg(y_td)&fl=*,score&fq=id:1&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 11100265 INFO  
(TEST-TestFunctionQuery.testDegreeRads-seed#[CC3F0BBD3EEA3BFC]) [    ] 
o.a.s.c.S.Request [collection1]  webapp=null path=null 
params={q={!func}deg(y_td)&fl=*,score&fq=id:2&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 11100266 INFO  
(TEST-TestFunctionQuery.testDegreeRads-seed#[CC3F0BBD3EEA3BFC]) [    ] 
o.a.s.c.S.Request [collection1]  webapp=null path=null 
params={q={!func}deg(y_td)&fl=*,score&fq=id:3&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 11100266 INFO  
(TEST-TestFunctionQuery.testDegreeRads-seed#[CC3F0BBD3EEA3BFC]) [    ] 
o.a.s.SolrTestCaseJ4 ###Ending testDegreeRads
   [junit4]   2> 11100267 INFO  
(SUITE-TestFunctionQuery-seed#[CC3F0BBD3EEA3BFC]-worker) [    ] 
o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> 11100267 INFO  
(SUITE-TestFunctionQuery-seed#[CC3F0BBD3EEA3BFC]-worker) [    ] 
o.a.s.c.CoreContainer Shutting down CoreContainer instance=920391407
   [junit4]   2> 11100267 INFO  
(SUITE-TestFunctionQuery-seed#[CC3F0BBD3EEA3BFC]-worker) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, 
tag=null
   [junit4]   2> 11100267 INFO  
(SUITE-TestFunctionQuery-seed#[CC3F0BBD3EEA3BFC]-worker) [    ] 
o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@15c48775: rootName = null, 
domain = solr.node, service url = null, agent id = null] for registry solr.node 
/ com.codahale.metrics.MetricRegistry@6f2a8504
   [junit4]   2> 11100286 INFO  
(SUITE-TestFunctionQuery-seed#[CC3F0BBD3EEA3BFC]-worker) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, 
tag=null
   [junit4]   2> 11100286 INFO  
(SUITE-TestFunctionQuery-seed#[CC3F0BBD3EEA3BFC]-worker) [    ] 
o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@6d195526: rootName = null, 
domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / 
com.codahale.metrics.MetricRegistry@3383f791
   [junit4]   2> 11100321 INFO  
(SUITE-TestFunctionQuery-seed#[CC3F0BBD3EEA3BFC]-worker) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, 
tag=null
   [junit4]   2> 11100321 INFO  
(SUITE-TestFunctionQuery-seed#[CC3F0BBD3EEA3BFC]-worker) [    ] 
o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@6badca6f: rootName = null, 
domain = solr.jetty, service url = null, agent id = null] for registry 
solr.jetty / com.codahale.metrics.MetricRegistry@5e3ab61a
   [junit4]   2> 11100323 INFO  (coreCloseExecutor-1876-thread-1) [    
x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore 
org.apache.solr.core.SolrCore@6a74442f
   [junit4]   2> 11100323 INFO  (coreCloseExecutor-1876-thread-1) [    
x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for 
registry=solr.core.collection1, tag=1786004527
   [junit4]   2> 11100323 INFO  (coreCloseExecutor-1876-thread-1) [    
x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@9b727b2: rootName = null, 
domain = solr.core.collection1, service url = null, agent id = null] for 
registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@17f15fe0
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.search.function.TestFunctionQuery_CC3F0BBD3EEA3BFC-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): 
{const_s=PostingsFormat(name=LuceneFixedGap), 
eff_tint=PostingsFormat(name=Direct), 1_s=PostingsFormat(name=Direct), 
foo_s=Lucene50(blocksize=128), 
x_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))),
 1-1_s=PostingsFormat(name=Direct), 
a_tfidf=PostingsFormat(name=LuceneFixedGap), 
field4_t=PostingsFormat(name=LuceneFixedGap), 
field2_t=PostingsFormat(name=Direct), field1_t=Lucene50(blocksize=128), 
amounts_dpf=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))),
 sfile_s=PostingsFormat(name=LuceneFixedGap), a_t=Lucene50(blocksize=128), 
id=PostingsFormat(name=LuceneFixedGap), text=PostingsFormat(name=Direct), 
vals_dpf=PostingsFormat(name=Direct)}, 
docValues:{eff_tint=DocValuesFormat(name=Asserting), 
a_i=DocValuesFormat(name=Memory), y_td=DocValuesFormat(name=Lucene70), 
foo_tl=DocValuesFormat(name=Asserting), foo_ti=DocValuesFormat(name=Direct), 
foo_tf=DocValuesFormat(name=Memory), sfile_s=DocValuesFormat(name=Memory), 
x:x_i=DocValuesFormat(name=Direct), id=DocValuesFormat(name=Memory), 
yak_i=DocValuesFormat(name=Asserting), text=DocValuesFormat(name=Asserting), 
vals_dpf=DocValuesFormat(name=Asserting), 
default_f=DocValuesFormat(name=Lucene70), x_td=DocValuesFormat(name=Memory), 
x_i=DocValuesFormat(name=Lucene70), a_tdt=DocValuesFormat(name=Lucene70), 
number_of_atoms_in_universe_l=DocValuesFormat(name=Asserting), 
foo_s=DocValuesFormat(name=Lucene70), x_s=DocValuesFormat(name=Direct), 
age_i=DocValuesFormat(name=Asserting), CoMpleX " fieldName 
_f=DocValuesFormat(name=Direct), 
default_amount_f=DocValuesFormat(name=Asserting), 
id_i=DocValuesFormat(name=Asserting), b_tdt=DocValuesFormat(name=Memory), 
foo_d=DocValuesFormat(name=Memory), foo_f=DocValuesFormat(name=Asserting)}, 
maxPointsInLeafNode=2020, maxMBSortInHeap=5.712847659330417, 
sim=RandomSimilarity(queryNorm=true): {}, locale=tr, timezone=America/Guyana
   [junit4]   2> NOTE: Linux 4.4.0-83-generic amd64/Oracle Corporation 
1.8.0_144 (64-bit)/cpus=4,threads=1,free=146637288,total=512753664
   [junit4]   2> NOTE: All tests run in this JVM: [XmlUpdateRequestHandlerTest, 
HdfsAutoAddReplicasIntegrationTest, DistanceFunctionTest, 
AsyncCallRequestStatusResponseTest, SolrCoreCheckLockOnStartupTest, 
TestRealTimeGet, RuleEngineTest, TestObjectReleaseTracker, 
DisMaxRequestHandlerTest, DefaultValueUpdateProcessorTest, 
TestEmbeddedSolrServerConstructors, BasicDistributedZk2Test, 
XsltUpdateRequestHandlerTest, AutoScalingHandlerTest, 
TestPayloadCheckQParserPlugin, TestReqParamsAPI, TestNumericTokenStream, 
TestTrackingShardHandlerFactory, TestFuzzyAnalyzedSuggestions, 
TestCSVResponseWriter, SystemInfoHandlerTest, TestSolrQueryResponse, 
TestCustomSort, AliasIntegrationTest, SpellPossibilityIteratorTest, 
TestSearchPerf, TestFieldCollectionResource, TestRequestStatusCollectionAPI, 
TestStreamBody, TestLeaderElectionWithEmptyReplica, 
CdcrReplicationDistributedZkTest, FullSolrCloudDistribCmdsTest, 
PropertiesRequestHandlerTest, HdfsWriteToMultipleCollectionsTest, 
TestLegacyFieldReuse, TestNamedUpdateProcessors, TestLegacyFieldCache, 
TestPartialUpdateDeduplication, TestRecoveryHdfs, DeleteStatusTest, 
LoggingHandlerTest, SolrIndexMetricsTest, TestLMJelinekMercerSimilarityFactory, 
AutoscalingHistoryHandlerTest, TestSQLHandlerNonCloud, SaslZkACLProviderTest, 
SortByFunctionTest, DeleteNodeTest, TestSolrCoreParser, 
DistributedSpellCheckComponentTest, TestFieldCacheSortRandom, RankQueryTest, 
HdfsThreadLeakTest, TestFunctionQuery]
   [junit4] Completed [676/745 (5!)] on J2 in 16.88s, 22 tests, 1 error <<< 
FAILURES!

[...truncated 229 lines...]
   [junit4] JVM J2: stdout was not empty, see: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/temp/junit4-J2-20171029_060056_6202382475134081855563.sysout
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/heapdumps/java_pid8437.hprof
 ...
   [junit4] Heap dump file created [659664832 bytes in 87.102 secs]
   [junit4] <<< JVM J2: EOF ----

   [junit4] JVM J2: stderr was not empty, see: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/temp/junit4-J2-20171029_060056_6201043745079792104157.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] 
   [junit4] Exception: java.lang.IllegalArgumentException thrown from the 
UncaughtExceptionHandler in thread "Connection evictor"
   [junit4] <<< JVM J2: EOF ----

[...truncated 8455 lines...]
BUILD FAILED
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/build.xml:834:
 The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/build.xml:778:
 Some of the tests produced a heap dump, but did not fail. Maybe a suppressed 
OutOfMemoryError? Dumps created:
* java_pid8437.hprof

Total time: 865 minutes 52 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to