Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/61/

10 tests failed.
FAILED:  
org.apache.lucene.codecs.memory.TestMemoryDocValuesFormat.testBinaryFixedLengthVsStoredFields

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([6CB5EC62AC1F849B]:0)


FAILED:  
junit.framework.TestSuite.org.apache.lucene.codecs.memory.TestMemoryDocValuesFormat

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([6CB5EC62AC1F849B]:0)


FAILED:  org.apache.lucene.index.TestIndexWriterOnVMError.testCheckpoint

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([B214C1465D2D4E3]:0)


FAILED:  
junit.framework.TestSuite.org.apache.lucene.index.TestIndexWriterOnVMError

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([B214C1465D2D4E3]:0)


FAILED:  org.apache.solr.cloud.RecoveryZkTest.test

Error Message:


Stack Trace:
java.util.concurrent.TimeoutException
        at 
__randomizedtesting.SeedInfo.seed([CEDB9D92D513606A:468FA2487BEF0D92]:0)
        at 
org.apache.solr.common.cloud.ZkStateReader.waitForState(ZkStateReader.java:1323)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.waitForState(CloudSolrClient.java:438)
        at org.apache.solr.cloud.RecoveryZkTest.test(RecoveryZkTest.java:122)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.TestConfigReload

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.handler.TestConfigReload:   
  1) Thread[id=11994, name=zkCallback-1410-thread-1, state=TIMED_WAITING, 
group=TGRP-TestConfigReload]         at sun.misc.Unsafe.park(Native Method)     
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)   
      at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
        at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE 
scope at org.apache.solr.handler.TestConfigReload: 
   1) Thread[id=11994, name=zkCallback-1410-thread-1, state=TIMED_WAITING, 
group=TGRP-TestConfigReload]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([CEDB9D92D513606A]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.TestConfigReload

Error Message:
There are still zombie threads that couldn't be terminated:    1) 
Thread[id=11994, name=zkCallback-1410-thread-1, state=TIMED_WAITING, 
group=TGRP-TestConfigReload]         at sun.misc.Unsafe.park(Native Method)     
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)   
      at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
        at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie 
threads that couldn't be terminated:
   1) Thread[id=11994, name=zkCallback-1410-thread-1, state=TIMED_WAITING, 
group=TGRP-TestConfigReload]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([CEDB9D92D513606A]:0)


FAILED:  
org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testNodeMarkersRegistration

Error Message:
Path /autoscaling/nodeAdded/127.0.0.1:60529_solr wasn't created

Stack Trace:
java.lang.AssertionError: Path /autoscaling/nodeAdded/127.0.0.1:60529_solr 
wasn't created
        at 
__randomizedtesting.SeedInfo.seed([CEDB9D92D513606A:D661159EDB26AD85]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at 
org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testNodeMarkersRegistration(TriggerIntegrationTest.java:909)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.HttpPartitionTest.test

Error Message:
Didn't see all replicas for shard shard1 in c8n_1x2 come up within 90000 ms! 
ClusterState: {   "collMinRf_1x3":{     "pullReplicas":"0",     
"replicationFactor":"1",     "shards":{"shard1":{         
"range":"80000000-7fffffff",         "state":"active",         "replicas":{     
      "core_node4":{             "core":"collMinRf_1x3_shard1_replica_n2",      
       "base_url":"http://127.0.0.1:35739";,             
"node_name":"127.0.0.1:35739_",             "state":"active",             
"type":"NRT"},           "core_node5":{             
"core":"collMinRf_1x3_shard1_replica_n3",             
"base_url":"http://127.0.0.1:56687";,             
"node_name":"127.0.0.1:56687_",             "state":"active",             
"type":"NRT",             "leader":"true"},           "core_node6":{            
 "core":"collMinRf_1x3_shard1_replica_n1",             
"base_url":"http://127.0.0.1:52172";,             
"node_name":"127.0.0.1:52172_",             "state":"active",             
"type":"NRT"}}}},     "router":{"name":"compositeId"},     
"maxShardsPerNode":"1",     "autoAddReplicas":"false",     "nrtReplicas":"3",   
  "tlogReplicas":"0"},   "collection1":{     "pullReplicas":"0",     
"replicationFactor":"1",     "shards":{       "shard1":{         
"range":"80000000-ffffffff",         "state":"active",         
"replicas":{"core_node44":{             
"core":"collection1_shard1_replica_n43",             
"base_url":"http://127.0.0.1:53968";,             
"node_name":"127.0.0.1:53968_",             "state":"active",             
"type":"NRT",             "leader":"true"}}},       "shard2":{         
"range":"0-7fffffff",         "state":"active",         "replicas":{           
"core_node42":{             "core":"collection1_shard2_replica_n41",            
 "base_url":"http://127.0.0.1:56687";,             
"node_name":"127.0.0.1:56687_",             "state":"active",             
"type":"NRT",             "leader":"true"},           "core_node46":{           
  "core":"collection1_shard2_replica_n45",             
"base_url":"http://127.0.0.1:35739";,             
"node_name":"127.0.0.1:35739_",             "state":"active",             
"type":"NRT"}}}},     "router":{"name":"compositeId"},     
"maxShardsPerNode":"1",     "autoAddReplicas":"false",     "nrtReplicas":"1",   
  "tlogReplicas":"0"},   "control_collection":{     "pullReplicas":"0",     
"replicationFactor":"1",     "shards":{"shard1":{         
"range":"80000000-7fffffff",         "state":"active",         
"replicas":{"core_node2":{             
"core":"control_collection_shard1_replica_n1",             
"base_url":"http://127.0.0.1:52172";,             
"node_name":"127.0.0.1:52172_",             "state":"active",             
"type":"NRT",             "leader":"true"}}}},     
"router":{"name":"compositeId"},     "maxShardsPerNode":"1",     
"autoAddReplicas":"false",     "nrtReplicas":"1",     "tlogReplicas":"0"},   
"c8n_1x2":{     "pullReplicas":"0",     "replicationFactor":"1",     
"shards":{"shard1":{         "range":"80000000-7fffffff",         
"state":"active",         "replicas":{           "core_node3":{             
"state":"down",             "base_url":"http://127.0.0.1:53968";,             
"core":"c8n_1x2_shard1_replica_n1",             "node_name":"127.0.0.1:53968_", 
            "type":"NRT"},           "core_node4":{             
"core":"c8n_1x2_shard1_replica_n2",             
"base_url":"http://127.0.0.1:56687";,             
"node_name":"127.0.0.1:56687_",             "state":"active",             
"type":"NRT",             "leader":"true"}}}},     
"router":{"name":"compositeId"},     "maxShardsPerNode":"1",     
"autoAddReplicas":"false",     "nrtReplicas":"2",     "tlogReplicas":"0"}}

Stack Trace:
java.lang.AssertionError: Didn't see all replicas for shard shard1 in c8n_1x2 
come up within 90000 ms! ClusterState: {
  "collMinRf_1x3":{
    "pullReplicas":"0",
    "replicationFactor":"1",
    "shards":{"shard1":{
        "range":"80000000-7fffffff",
        "state":"active",
        "replicas":{
          "core_node4":{
            "core":"collMinRf_1x3_shard1_replica_n2",
            "base_url":"http://127.0.0.1:35739";,
            "node_name":"127.0.0.1:35739_",
            "state":"active",
            "type":"NRT"},
          "core_node5":{
            "core":"collMinRf_1x3_shard1_replica_n3",
            "base_url":"http://127.0.0.1:56687";,
            "node_name":"127.0.0.1:56687_",
            "state":"active",
            "type":"NRT",
            "leader":"true"},
          "core_node6":{
            "core":"collMinRf_1x3_shard1_replica_n1",
            "base_url":"http://127.0.0.1:52172";,
            "node_name":"127.0.0.1:52172_",
            "state":"active",
            "type":"NRT"}}}},
    "router":{"name":"compositeId"},
    "maxShardsPerNode":"1",
    "autoAddReplicas":"false",
    "nrtReplicas":"3",
    "tlogReplicas":"0"},
  "collection1":{
    "pullReplicas":"0",
    "replicationFactor":"1",
    "shards":{
      "shard1":{
        "range":"80000000-ffffffff",
        "state":"active",
        "replicas":{"core_node44":{
            "core":"collection1_shard1_replica_n43",
            "base_url":"http://127.0.0.1:53968";,
            "node_name":"127.0.0.1:53968_",
            "state":"active",
            "type":"NRT",
            "leader":"true"}}},
      "shard2":{
        "range":"0-7fffffff",
        "state":"active",
        "replicas":{
          "core_node42":{
            "core":"collection1_shard2_replica_n41",
            "base_url":"http://127.0.0.1:56687";,
            "node_name":"127.0.0.1:56687_",
            "state":"active",
            "type":"NRT",
            "leader":"true"},
          "core_node46":{
            "core":"collection1_shard2_replica_n45",
            "base_url":"http://127.0.0.1:35739";,
            "node_name":"127.0.0.1:35739_",
            "state":"active",
            "type":"NRT"}}}},
    "router":{"name":"compositeId"},
    "maxShardsPerNode":"1",
    "autoAddReplicas":"false",
    "nrtReplicas":"1",
    "tlogReplicas":"0"},
  "control_collection":{
    "pullReplicas":"0",
    "replicationFactor":"1",
    "shards":{"shard1":{
        "range":"80000000-7fffffff",
        "state":"active",
        "replicas":{"core_node2":{
            "core":"control_collection_shard1_replica_n1",
            "base_url":"http://127.0.0.1:52172";,
            "node_name":"127.0.0.1:52172_",
            "state":"active",
            "type":"NRT",
            "leader":"true"}}}},
    "router":{"name":"compositeId"},
    "maxShardsPerNode":"1",
    "autoAddReplicas":"false",
    "nrtReplicas":"1",
    "tlogReplicas":"0"},
  "c8n_1x2":{
    "pullReplicas":"0",
    "replicationFactor":"1",
    "shards":{"shard1":{
        "range":"80000000-7fffffff",
        "state":"active",
        "replicas":{
          "core_node3":{
            "state":"down",
            "base_url":"http://127.0.0.1:53968";,
            "core":"c8n_1x2_shard1_replica_n1",
            "node_name":"127.0.0.1:53968_",
            "type":"NRT"},
          "core_node4":{
            "core":"c8n_1x2_shard1_replica_n2",
            "base_url":"http://127.0.0.1:56687";,
            "node_name":"127.0.0.1:56687_",
            "state":"active",
            "type":"NRT",
            "leader":"true"}}}},
    "router":{"name":"compositeId"},
    "maxShardsPerNode":"1",
    "autoAddReplicas":"false",
    "nrtReplicas":"2",
    "tlogReplicas":"0"}}
        at 
__randomizedtesting.SeedInfo.seed([CEDB9D92D513606A:468FA2487BEF0D92]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.ensureAllReplicasAreActive(AbstractFullDistribZkTestBase.java:1996)
        at 
org.apache.solr.cloud.HttpPartitionTest.testRf2(HttpPartitionTest.java:370)
        at 
org.apache.solr.cloud.HttpPartitionTest.test(HttpPartitionTest.java:132)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest.test

Error Message:
There are still nodes recoverying - waited for 600 seconds

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 600 
seconds
        at 
__randomizedtesting.SeedInfo.seed([CEDB9D92D513606A:468FA2487BEF0D92]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:185)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:914)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1471)
        at 
org.apache.solr.cloud.TlogReplayBufferedWhileIndexingTest.test(TlogReplayBufferedWhileIndexingTest.java:99)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 1650 lines...]
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterOnVMError
   [junit4]   2> Oct 13, 2017 12:43:54 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
   [junit4]   2> WARNING: Suite execution timed out: 
org.apache.lucene.index.TestIndexWriterOnVMError
   [junit4]   2>    1) Thread[id=9, name=JUnit4-serializer-daemon, 
state=TIMED_WAITING, group=main]
   [junit4]   2>         at java.lang.Thread.sleep(Native Method)
   [junit4]   2>         at 
com.carrotsearch.ant.tasks.junit4.events.Serializer$1.run(Serializer.java:50)
   [junit4]   2>    2) Thread[id=1, name=main, state=WAITING, group=main]
   [junit4]   2>         at java.lang.Object.wait(Native Method)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1252)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1326)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:636)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:493)
   [junit4]   2>         at 
com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:251)
   [junit4]   2>         at 
com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:368)
   [junit4]   2>         at 
com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
   [junit4]   2>    3) Thread[id=2447, 
name=SUITE-TestIndexWriterOnVMError-seed#[B214C1465D2D4E3], state=RUNNABLE, 
group=TGRP-TestIndexWriterOnVMError]
   [junit4]   2>         at java.lang.Thread.getStackTrace(Thread.java:1559)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:696)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:693)
   [junit4]   2>         at java.security.AccessController.doPrivileged(Native 
Method)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.getStackTrace(ThreadLeakControl.java:693)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.getThreadsWithTraces(ThreadLeakControl.java:709)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:689)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:65)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:415)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:705)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:139)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:626)
   [junit4]   2>    4) Thread[id=2448, 
name=TEST-TestIndexWriterOnVMError.testCheckpoint-seed#[B214C1465D2D4E3], 
state=RUNNABLE, group=TGRP-TestIndexWriterOnVMError]
   [junit4]   2>         at java.lang.Throwable.getStackTraceElement(Native 
Method)
   [junit4]   2>         at 
java.lang.Throwable.getOurStackTrace(Throwable.java:827)
   [junit4]   2>         at 
java.lang.Throwable.getStackTrace(Throwable.java:816)
   [junit4]   2>         at java.lang.Thread.getStackTrace(Thread.java:1559)
   [junit4]   2>         at 
org.apache.lucene.index.TestIndexWriterOnVMError$4.eval(TestIndexWriterOnVMError.java:283)
   [junit4]   2>         at 
org.apache.lucene.store.MockDirectoryWrapper.maybeThrowDeterministicException(MockDirectoryWrapper.java:1022)
   [junit4]   2>         at 
org.apache.lucene.store.MockIndexOutputWrapper.writeBytes(MockIndexOutputWrapper.java:144)
   [junit4]   2>         at 
org.apache.lucene.store.MockIndexOutputWrapper.writeByte(MockIndexOutputWrapper.java:126)
   [junit4]   2>         at 
org.apache.lucene.store.DataOutput.writeInt(DataOutput.java:70)
   [junit4]   2>         at 
org.apache.lucene.codecs.lucene70.Lucene70DocValuesConsumer.addSortedSetField(Lucene70DocValuesConsumer.java:607)
   [junit4]   2>         at 
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addSortedSetField(PerFieldDocValuesFormat.java:129)
   [junit4]   2>         at 
org.apache.lucene.index.SortedSetDocValuesWriter.flush(SortedSetDocValuesWriter.java:221)
   [junit4]   2>         at 
org.apache.lucene.index.DefaultIndexingChain.writeDocValues(DefaultIndexingChain.java:247)
   [junit4]   2>         at 
org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:131)
   [junit4]   2>         at 
org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:451)
   [junit4]   2>         at 
org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:542)
   [junit4]   2>         at 
org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:658)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3201)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3413)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3378)
   [junit4]   2>         at 
org.apache.lucene.index.TestIndexWriterOnVMError.doTest(TestIndexWriterOnVMError.java:181)
   [junit4]   2>         at 
org.apache.lucene.index.TestIndexWriterOnVMError.testCheckpoint(TestIndexWriterOnVMError.java:280)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
   [junit4]   2>         at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
   [junit4]   2>         at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 
'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=TestIndexWriterOnVMError -Dtests.method=testCheckpoint 
-Dtests.seed=B214C1465D2D4E3 -Dtests.multiplier=2 -Dtests.nightly=true 
-Dtests.slow=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt
 -Dtests.locale=en-CA -Dtests.timezone=AGT -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8
   [junit4] ERROR   7200s J0 | TestIndexWriterOnVMError.testCheckpoint <<<
   [junit4]    > Throwable #1: java.lang.Exception: Test abandoned because 
suite timeout was reached.
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([B214C1465D2D4E3]:0)
   [junit4]   2> Oct 13, 2017 12:43:55 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked 
thread(s).
   [junit4]   2> Oct 13, 2017 12:44:15 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at 
org.apache.lucene.index.TestIndexWriterOnVMError: 
   [junit4]   2>    1) Thread[id=2448, 
name=TEST-TestIndexWriterOnVMError.testCheckpoint-seed#[B214C1465D2D4E3], 
state=RUNNABLE, group=TGRP-TestIndexWriterOnVMError]
   [junit4]   2>         at java.lang.Throwable.getStackTraceElement(Native 
Method)
   [junit4]   2>         at 
java.lang.Throwable.getOurStackTrace(Throwable.java:827)
   [junit4]   2>         at 
java.lang.Throwable.getStackTrace(Throwable.java:816)
   [junit4]   2>         at java.lang.Thread.getStackTrace(Thread.java:1559)
   [junit4]   2>         at 
org.apache.lucene.index.TestIndexWriterOnVMError$4.eval(TestIndexWriterOnVMError.java:283)
   [junit4]   2>         at 
org.apache.lucene.store.MockDirectoryWrapper.maybeThrowDeterministicException(MockDirectoryWrapper.java:1022)
   [junit4]   2>         at 
org.apache.lucene.store.MockIndexOutputWrapper.writeBytes(MockIndexOutputWrapper.java:144)
   [junit4]   2>         at 
org.apache.lucene.store.MockIndexOutputWrapper.writeByte(MockIndexOutputWrapper.java:126)
   [junit4]   2>         at 
org.apache.lucene.codecs.lucene70.Lucene70DocValuesConsumer.addTermsDict(Lucene70DocValuesConsumer.java:465)
   [junit4]   2>         at 
org.apache.lucene.codecs.lucene70.Lucene70DocValuesConsumer.addSortedSetField(Lucene70DocValuesConsumer.java:627)
   [junit4]   2>         at 
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addSortedSetField(PerFieldDocValuesFormat.java:129)
   [junit4]   2>         at 
org.apache.lucene.index.SortedSetDocValuesWriter.flush(SortedSetDocValuesWriter.java:221)
   [junit4]   2>         at 
org.apache.lucene.index.DefaultIndexingChain.writeDocValues(DefaultIndexingChain.java:247)
   [junit4]   2>         at 
org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:131)
   [junit4]   2>         at 
org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:451)
   [junit4]   2>         at 
org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:542)
   [junit4]   2>         at 
org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:658)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:453)
   [junit4]   2>         at 
org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:103)
   [junit4]   2>         at 
org.apache.lucene.index.TestIndexWriterOnVMError.doTest(TestIndexWriterOnVMError.java:175)
   [junit4]   2>         at 
org.apache.lucene.index.TestIndexWriterOnVMError.testCheckpoint(TestIndexWriterOnVMError.java:280)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
   [junit4]   2>         at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
   [junit4]   2>         at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Oct 13, 2017 12:44:15 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=2448, 
name=TEST-TestIndexWriterOnVMError.testCheckpoint-seed#[B214C1465D2D4E3], 
state=RUNNABLE, group=TGRP-TestIndexWriterOnVMError]
   [junit4]   2> Oct 13, 2017 12:44:19 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> SEVERE: There are still zombie threads that couldn't be 
terminated:
   [junit4]   2>    1) Thread[id=2448, 
name=TEST-TestIndexWriterOnVMError.testCheckpoint-seed#[B214C1465D2D4E3], 
state=RUNNABLE, group=TGRP-TestIndexWriterOnVMError]
   [junit4]   2>         at java.lang.Throwable.getStackTraceElement(Native 
Method)
   [junit4]   2>         at 
java.lang.Throwable.getOurStackTrace(Throwable.java:827)
   [junit4]   2>         at 
java.lang.Throwable.getStackTrace(Throwable.java:816)
   [junit4]   2>         at java.lang.Thread.getStackTrace(Thread.java:1559)
   [junit4]   2>         at 
org.apache.lucene.index.TestIndexWriterOnVMError$4.eval(TestIndexWriterOnVMError.java:283)
   [junit4]   2>         at 
org.apache.lucene.store.MockDirectoryWrapper.maybeThrowDeterministicException(MockDirectoryWrapper.java:1022)
   [junit4]   2>         at 
org.apache.lucene.store.MockIndexOutputWrapper.writeBytes(MockIndexOutputWrapper.java:144)
   [junit4]   2>         at 
org.apache.lucene.store.MockIndexOutputWrapper.writeByte(MockIndexOutputWrapper.java:126)
   [junit4]   2>         at 
org.apache.lucene.store.DataOutput.writeInt(DataOutput.java:71)
   [junit4]   2>         at 
org.apache.lucene.store.DataOutput.writeLong(DataOutput.java:212)
   [junit4]   2>         at 
org.apache.lucene.codecs.lucene70.Lucene70DocValuesConsumer.addSortedSetField(Lucene70DocValuesConsumer.java:590)
   [junit4]   2>         at 
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addSortedSetField(PerFieldDocValuesFormat.java:129)
   [junit4]   2>         at 
org.apache.lucene.index.SortedSetDocValuesWriter.flush(SortedSetDocValuesWriter.java:221)
   [junit4]   2>         at 
org.apache.lucene.index.DefaultIndexingChain.writeDocValues(DefaultIndexingChain.java:247)
   [junit4]   2>         at 
org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:131)
   [junit4]   2>         at 
org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:451)
   [junit4]   2>         at 
org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:542)
   [junit4]   2>         at 
org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:658)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3201)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3413)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3378)
   [junit4]   2>         at 
org.apache.lucene.index.TestIndexWriterOnVMError.doTest(TestIndexWriterOnVMError.java:181)
   [junit4]   2>         at 
org.apache.lucene.index.TestIndexWriterOnVMError.testCheckpoint(TestIndexWriterOnVMError.java:280)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
   [junit4]   2>         at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
   [junit4]   2>         at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): 
{text_payloads=BlockTreeOrds(blocksize=128), 
text_vectors=PostingsFormat(name=Asserting), text1=PostingsFormat(name=Direct), 
id=Lucene50(blocksize=128)}, docValues:{dv3=DocValuesFormat(name=Direct), 
dv2=DocValuesFormat(name=Asserting), dv5=DocValuesFormat(name=Lucene70), 
dv=DocValuesFormat(name=Lucene70), text_payloads=DocValuesFormat(name=Direct), 
dv4=DocValuesFormat(name=Lucene70), 
text_vectors=DocValuesFormat(name=Lucene70), 
text1=DocValuesFormat(name=Asserting), id=DocValuesFormat(name=Lucene70), 
point2d=DocValuesFormat(name=Lucene70), stored1=DocValuesFormat(name=Lucene70), 
point=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=1892, 
maxMBSortInHeap=7.141004137114521, sim=RandomSimilarity(queryNorm=false): 
{text_payloads=DFR I(F)L2, text_vectors=DFR GB2, text1=IB SPL-D2}, 
locale=en-CA, timezone=AGT
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 
1.8.0_144 (64-bit)/cpus=4,threads=2,free=143495048,total=526909440
   [junit4]   2> NOTE: All tests run in this JVM: [TestPagedBytes, 
TestPerFieldPostingsFormat, TestSloppyMath, TestGeoUtils, 
TestDeterminizeLexicon, TestMaxTermFrequency, TestSimilarity, TestTermsEnum, 
TestBagOfPositions, TestIndexingSequenceNumbers, TestSort, TestSearchAfter, 
TestStandardAnalyzer, TestSearch, Test2BTerms, TestTotalHitCountCollector, 
TestCodecs, TestBlockPostingsFormat2, TestManyFields, TestThreadedForceMerge, 
TestRegexpRandom, TestCrashCausesCorruptIndex, TestIntBlockPool, 
TestCharFilter, TestNIOFSDirectory, TestSnapshotDeletionPolicy, TestBytesRef, 
TestIndexWriterOnVMError]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 
'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=TestIndexWriterOnVMError -Dtests.seed=B214C1465D2D4E3 
-Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt
 -Dtests.locale=en-CA -Dtests.timezone=AGT -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J0 | TestIndexWriterOnVMError (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 
7200000 msec).
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([B214C1465D2D4E3]:0)
   [junit4] Completed [311/455 (1!)] on J0 in 7225.17s, 1 test, 2 errors <<< 
FAILURES!

[...truncated 5776 lines...]
   [junit4] Suite: org.apache.lucene.codecs.memory.TestMemoryDocValuesFormat
   [junit4]   2> 10? 13, 2017 9:42:52 ?? 
com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
   [junit4]   2> WARNING: Suite execution timed out: 
org.apache.lucene.codecs.memory.TestMemoryDocValuesFormat
   [junit4]   2>    1) Thread[id=9, name=JUnit4-serializer-daemon, 
state=TIMED_WAITING, group=main]
   [junit4]   2>         at java.lang.Thread.sleep(Native Method)
   [junit4]   2>         at 
com.carrotsearch.ant.tasks.junit4.events.Serializer$1.run(Serializer.java:50)
   [junit4]   2>    2) Thread[id=1, name=main, state=WAITING, group=main]
   [junit4]   2>         at java.lang.Object.wait(Native Method)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1252)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1326)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:636)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:493)
   [junit4]   2>         at 
com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:251)
   [junit4]   2>         at 
com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:368)
   [junit4]   2>         at 
com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
   [junit4]   2>    3) Thread[id=255, 
name=SUITE-TestMemoryDocValuesFormat-seed#[6CB5EC62AC1F849B], state=RUNNABLE, 
group=TGRP-TestMemoryDocValuesFormat]
   [junit4]   2>         at java.lang.Thread.getStackTrace(Thread.java:1559)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:696)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:693)
   [junit4]   2>         at java.security.AccessController.doPrivileged(Native 
Method)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.getStackTrace(ThreadLeakControl.java:693)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.getThreadsWithTraces(ThreadLeakControl.java:709)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:689)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:65)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:415)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:705)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:139)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:626)
   [junit4]   2>    4) Thread[id=256, 
name=TEST-TestMemoryDocValuesFormat.testBinaryFixedLengthVsStoredFields-seed#[6CB5EC62AC1F849B],
 state=TIMED_WAITING, group=TGRP-TestMemoryDocValuesFormat]
   [junit4]   2>         at java.lang.Object.wait(Native Method)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.doWait(IndexWriter.java:4628)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:2091)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:2019)
   [junit4]   2>         at 
org.apache.lucene.index.RandomIndexWriter.forceMerge(RandomIndexWriter.java:429)
   [junit4]   2>         at 
org.apache.lucene.index.BaseDocValuesFormatTestCase.doTestBinaryVsStoredFields(BaseDocValuesFormatTestCase.java:1472)
   [junit4]   2>         at 
org.apache.lucene.index.BaseDocValuesFormatTestCase.doTestBinaryFixedLengthVsStoredFields(BaseDocValuesFormatTestCase.java:1508)
   [junit4]   2>         at 
org.apache.lucene.index.BaseDocValuesFormatTestCase.testBinaryFixedLengthVsStoredFields(BaseDocValuesFormatTestCase.java:1497)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
   [junit4]   2>         at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
   [junit4]   2>         at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 
'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=TestMemoryDocValuesFormat 
-Dtests.method=testBinaryFixedLengthVsStoredFields 
-Dtests.seed=6CB5EC62AC1F849B -Dtests.multiplier=2 -Dtests.nightly=true 
-Dtests.slow=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt
 -Dtests.locale=ko-KR -Dtests.timezone=Atlantic/Madeira -Dtests.asserts=true 
-Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   7094s J0 | 
TestMemoryDocValuesFormat.testBinaryFixedLengthVsStoredFields <<<
   [junit4]    > Throwable #1: java.lang.Exception: Test abandoned because 
suite timeout was reached.
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([6CB5EC62AC1F849B]:0)
   [junit4]   2> 10? 13, 2017 9:42:52 ?? 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked 
thread(s).
   [junit4]   2> 10? 13, 2017 9:43:12 ?? 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at 
org.apache.lucene.codecs.memory.TestMemoryDocValuesFormat: 
   [junit4]   2>    1) Thread[id=256, 
name=TEST-TestMemoryDocValuesFormat.testBinaryFixedLengthVsStoredFields-seed#[6CB5EC62AC1F849B],
 state=TIMED_WAITING, group=TGRP-TestMemoryDocValuesFormat]
   [junit4]   2>         at java.lang.Object.wait(Native Method)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.doWait(IndexWriter.java:4628)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:2091)
   [junit4]   2>         at 
org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:2019)
   [junit4]   2>         at 
org.apache.lucene.index.RandomIndexWriter.forceMerge(RandomIndexWriter.java:429)
   [junit4]   2>         at 
org.apache.lucene.index.BaseDocValuesFormatTestCase.doTestBinaryVsStoredFields(BaseDocValuesFormatTestCase.java:1472)
   [junit4]   2>         at 
org.apache.lucene.index.BaseDocValuesFormatTestCase.doTestBinaryFixedLengthVsStoredFields(BaseDocValuesFormatTestCase.java:1508)
   [junit4]   2>         at 
org.apache.lucene.index.BaseDocValuesFormatTestCase.testBinaryFixedLengthVsStoredFields(BaseDocValuesFormatTestCase.java:1497)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
   [junit4]   2>         at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
   [junit4]   2>         at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 10? 13, 2017 9:43:12 ?? 
com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=256, 
name=TEST-TestMemoryDocValuesFormat.testBinaryFixedLengthVsStoredFields-seed#[6CB5EC62AC1F849B],
 state=TIMED_WAITING, group=TGRP-TestMemoryDocValuesFormat]
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/build/codecs/test/J0/temp/lucene.codecs.memory.TestMemoryDocValuesFormat_6CB5EC62AC1F849B-001
   [junit4]   2> Oct 13, 2017 8:43:12 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: All leaked threads terminated.
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, 
docValues:{}, maxPointsInLeafNode=628, maxMBSortInHeap=6.493987228103301, 
sim=RandomSimilarity(queryNorm=true): {fieldname=LM Jelinek-Mercer(0.100000)}, 
locale=ko-KR, timezone=Atlantic/Madeira
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 
1.8.0_144 (64-bit)/cpus=4,threads=1,free=224596248,total=408944640
   [junit4]   2> NOTE: All tests run in this JVM: 
[TestSimpleTextTermVectorsFormat, TestSimpleTextStoredFieldsFormat, 
TestMemoryDocValuesFormat]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 
'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=TestMemoryDocValuesFormat -Dtests.seed=6CB5EC62AC1F849B 
-Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt
 -Dtests.locale=ko-KR -Dtests.timezone=Atlantic/Madeira -Dtests.asserts=true 
-Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J0 | TestMemoryDocValuesFormat (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 
7200000 msec).
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([6CB5EC62AC1F849B]:0)
   [junit4] Completed [20/20 (1!)] on J0 in 7220.43s, 6 tests, 2 errors <<< 
FAILURES!

[...truncated 6044 lines...]
   [junit4] Suite: 
org.apache.solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest
   [junit4]   2> Creating dataDir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_CEDB9D92D513606A-001/init-core-data-001
   [junit4]   2> 4736900 WARN  
(SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[CEDB9D92D513606A]-worker) 
[    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=186 numCloses=186
   [junit4]   2> 4736901 INFO  
(SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[CEDB9D92D513606A]-worker) 
[    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) 
w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 4736902 INFO  
(SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[CEDB9D92D513606A]-worker) 
[    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: 
@org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=None)
   [junit4]   2> 4736902 INFO  
(SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[CEDB9D92D513606A]-worker) 
[    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: 
test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 4736902 INFO  
(SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[CEDB9D92D513606A]-worker) 
[    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: 
/
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 4737598 WARN  
(SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[CEDB9D92D513606A]-worker) 
[    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried 
hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 4737759 WARN  
(SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[CEDB9D92D513606A]-worker) 
[    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 4737838 INFO  
(SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[CEDB9D92D513606A]-worker) 
[    ] o.m.log jetty-6.1.26
   [junit4]   2> 4737981 INFO  
(SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[CEDB9D92D513606A]-worker) 
[    ] o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/hdfs
 to ./temp/Jetty_localhost_39195_hdfs____294p75/webapp
   [junit4]   2> 4742637 INFO  
(SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[CEDB9D92D513606A]-worker) 
[    ] o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:39195
   [junit4]   2> 4744923 WARN  
(SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[CEDB9D92D513606A]-worker) 
[    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 4744924 INFO  
(SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[CEDB9D92D513606A]-worker) 
[    ] o.m.log jetty-6.1.26
   [junit4]   2> 4744975 INFO  
(SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[CEDB9D92D513606A]-worker) 
[    ] o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode
 to ./temp/Jetty_localhost_60271_datanode____.qn6hs7/webapp
   [junit4]   2> 4747864 INFO  
(SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[CEDB9D92D513606A]-worker) 
[    ] o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:60271
   [junit4]   2> 4748939 WARN  
(SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[CEDB9D92D513606A]-worker) 
[    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 4748940 INFO  
(SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[CEDB9D92D513606A]-worker) 
[    ] o.m.log jetty-6.1.26
   [junit4]   2> 4749004 INFO  
(SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[CEDB9D92D513606A]-worker) 
[    ] o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode
 to ./temp/Jetty_localhost_37029_datanode____dbnr3u/webapp
   [junit4]   2> 4750765 ERROR (DataNode: 
[[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_CEDB9D92D513606A-001/tempDir-001/hdfsBaseDir/data/data1/,
 
[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_CEDB9D92D513606A-001/tempDir-001/hdfsBaseDir/data/data2/]]
  heartbeating to localhost/127.0.0.1:50925) [    ] 
o.a.h.h.s.d.DirectoryScanner 
dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 
ms/sec. Assuming default value of 1000
   [junit4]   2> 4751487 INFO  (Block report processor) [    ] BlockStateChange 
BLOCK* processReport 0x1d1d7fe9a60803: from storage 
DS-385cf686-25f4-49d1-94b5-8992fcbac21a node 
DatanodeRegistration(127.0.0.1:57070, 
datanodeUuid=b8a78467-605a-4250-b1d3-d45d893495e6, infoPort=51939, 
infoSecurePort=0, ipcPort=55452, 
storageInfo=lv=-56;cid=testClusterID;nsid=797534868;c=0), blocks: 0, 
hasStaleStorage: true, processing time: 2 msecs
   [junit4]   2> 4751487 INFO  (Block report processor) [    ] BlockStateChange 
BLOCK* processReport 0x1d1d7fe9a60803: from storage 
DS-9b7d8b72-de5d-4977-b42f-fc441d26353a node 
DatanodeRegistration(127.0.0.1:57070, 
datanodeUuid=b8a78467-605a-4250-b1d3-d45d893495e6, infoPort=51939, 
infoSecurePort=0, ipcPort=55452, 
storageInfo=lv=-56;cid=testClusterID;nsid=797534868;c=0), blocks: 0, 
hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 4754307 INFO  
(SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[CEDB9D92D513606A]-worker) 
[    ] o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37029
   [junit4]   2> 4761056 ERROR (DataNode: 
[[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_CEDB9D92D513606A-001/tempDir-001/hdfsBaseDir/data/data3/,
 
[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_CEDB9D92D513606A-001/tempDir-001/hdfsBaseDir/data/data4/]]
  heartbeating to localhost/127.0.0.1:50925) [    ] 
o.a.h.h.s.d.DirectoryScanner 
dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 
ms/sec. Assuming default value of 1000
   [junit4]   2> 4761277 INFO  (Block report processor) [    ] BlockStateChange 
BLOCK* processReport 0x1d1d822f245e36: from storage 
DS-1784222d-50a1-4624-a7f6-636be5ca11a6 node 
DatanodeRegistration(127.0.0.1:41026, 
datanodeUuid=35111e7b-ac8b-456f-8d96-c1e943e88aae, infoPort=48390, 
infoSecurePort=0, ipcPort=35882, 
storageInfo=lv=-56;cid=testClusterID;nsid=797534868;c=0), blocks: 0, 
hasStaleStorage: true, processing time: 40 msecs
   [junit4]   2> 4761293 INFO  (Block report processor) [    ] BlockStateChange 
BLOCK* processReport 0x1d1d822f245e36: from storage 
DS-75b9b180-d5e7-4a9a-85db-060ac81cb0f7 node 
DatanodeRegistration(127.0.0.1:41026, 
datanodeUuid=35111e7b-ac8b-456f-8d96-c1e943e88aae, infoPort=48390, 
infoSecurePort=0, ipcPort=35882, 
storageInfo=lv=-56;cid=testClusterID;nsid=797534868;c=0), blocks: 0, 
hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 4761627 INFO  
(TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[CEDB9D92D513606A]) [   
 ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 4761627 INFO  (Thread-7136) [    ] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 4761627 INFO  (Thread-7136) [    ] o.a.s.c.ZkTestServer 
Starting server
   [junit4]   2> 4761671 ERROR (Thread-7136) [    ] o.a.z.s.ZooKeeperServer 
ZKShutdownHandler is not registered, so ZooKeeper server won't take any action 
on ERROR or SHUTDOWN server state changes
   [junit4]   2> 4761734 INFO  
(TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[CEDB9D92D513606A]) [   
 ] o.a.s.c.ZkTestServer start zk server on port:43730
   [junit4]   2> 4762503 INFO  
(TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[CEDB9D92D513606A]) [   
 ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
 to /configs/conf1/solrconfig.xml
   [junit4]   2> 4762504 INFO  
(TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[CEDB9D92D513606A]) [   
 ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml
 to /configs/conf1/schema.xml
   [junit4]   2> 4762505 INFO  
(TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[CEDB9D92D513606A]) [   
 ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
 to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 4762505 INFO  
(TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[CEDB9D92D513606A]) [   
 ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
 to /configs/conf1/stopwords.txt
   [junit4]   2> 4762523 INFO  
(TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[CEDB9D92D513606A]) [   
 ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt
 to /configs/conf1/protwords.txt
   [junit4]   2> 4762524 INFO  
(TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[CEDB9D92D513606A]) [   
 ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml
 to /configs/conf1/currency.xml
   [junit4]   2> 4762525 INFO  
(TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[CEDB9D92D513606A]) [   
 ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
 to /configs/conf1/enumsConfig.xml
   [junit4]   2> 4762525 INFO  
(TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[CEDB9D92D513606A]) [   
 ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
 to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 4762543 INFO  
(TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[CEDB9D92D513606A]) [   
 ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
 to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 4762544 INFO  
(TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[CEDB9D92D513606A]) [   
 ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
 to /configs/conf1/old_synonyms.txt
   [junit4]   2> 4762544 INFO  
(TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[CEDB9D92D513606A]) [   
 ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
 to /configs/conf1/synonyms.txt
   [junit4]   2> 4762584 INFO  
(TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[CEDB9D92D513606A]) [   
 ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless 
explicitly asked otherwise
   [junit4]   2> 4763765 INFO  
(TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[CEDB9D92D513606A]) [   
 ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 4763786 INFO  
(TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[CEDB9D92D513606A]) [   
 ] o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@24da635f{/,null,AVAILABLE}
   [junit4]   2> 4763786 INFO  
(TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[CEDB9D92D513606A]) [   
 ] o.e.j.s.AbstractConnector Started 
ServerConnector@4f6094ec{HTTP/1.1,[http/1.1]}{127.0.0.1:59235}
   [junit4]   2> 4763786 INFO  
(TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[CEDB9D92D513606A]) [   
 ] o.e.j.s.Server Started @4782523ms
   [junit4]   2> 4763786 INFO  
(TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[CEDB9D92D513606A]) [   
 ] o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=hdfs://localhost:50925/hdfs__localhost_50925__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-7.x_checkout_solr_build_solr-core_test_J1_temp_solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_CEDB9D92D513606A-001_tempDir-002_control_data,
 replicaType=NRT, hostContext=/, hostPort=59235, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_CEDB9D92D513606A-001/control-001/cores}
   [junit4]   2> 4763787 ERROR 
(TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[CEDB9D92D513606A]) [   
 ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 4763787 INFO  
(TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[CEDB9D92D513606A]) [   
 ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 
7.2.0
   [junit4]   2> 4763787 INFO  
(TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[CEDB9D92D513606A]) [   
 ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port 
null
   [junit4]   2> 4763787 INFO  
(TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[CEDB9D92D513606A]) [   
 ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default 
config dir: null
   [junit4]   2> 4763787 INFO  
(TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[CEDB9D92D513606A]) [   
 ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-10-14T01:08:15.

[...truncated too long message...]

leader, tag=1437374817
   [junit4]   2> 9391060 INFO  (jetty-closer-3680-thread-1) [    ] 
o.a.s.c.Overseer Overseer 
(id=98824866184232966-127.0.0.1:54042_solr-n_0000000001) closing
   [junit4]   2> 9391060 INFO  
(OverseerStateUpdate-98824866184232966-127.0.0.1:54042_solr-n_0000000001) 
[n:127.0.0.1:54042_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 
127.0.0.1:54042_solr
   [junit4]   2> 9391060 WARN  
(OverseerAutoScalingTriggerThread-98824866184232966-127.0.0.1:54042_solr-n_0000000001)
 [n:127.0.0.1:54042_solr    ] o.a.s.c.a.OverseerTriggerThread 
OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 9391073 WARN  
(zkCallback-3691-thread-5-processing-n:127.0.0.1:54042_solr) 
[n:127.0.0.1:54042_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, 
but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 9391074 INFO  (jetty-closer-3680-thread-1) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@231de5c6{/solr,null,UNAVAILABLE}
   [junit4]   2> 9391092 INFO  
(zkCallback-3702-thread-4-processing-n:127.0.0.1:57668_solr) 
[n:127.0.0.1:57668_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (2) -> (1)
   [junit4]   2> 9391092 INFO  
(zkCallback-3702-thread-5-processing-n:127.0.0.1:57668_solr) 
[n:127.0.0.1:57668_solr    ] o.a.s.c.OverseerElectionContext I am going to be 
the leader 127.0.0.1:57668_solr
   [junit4]   2> 9395821 INFO  
(recoveryExecutor-3700-thread-1-processing-n:127.0.0.1:57668_solr 
x:recoverytest_shard1_replica_n3 s:shard1 c:recoverytest r:core_node4) 
[n:127.0.0.1:57668_solr c:recoverytest s:shard1 r:core_node4 
x:recoverytest_shard1_replica_n3] o.a.s.c.RecoveryStrategy RecoveryStrategy has 
been closed
   [junit4]   2> 9395821 INFO  
(recoveryExecutor-3700-thread-1-processing-n:127.0.0.1:57668_solr 
x:recoverytest_shard1_replica_n3 s:shard1 c:recoverytest r:core_node4) 
[n:127.0.0.1:57668_solr c:recoverytest s:shard1 r:core_node4 
x:recoverytest_shard1_replica_n3] o.a.s.c.RecoveryStrategy Finished recovery 
process, successful=[false]
   [junit4]   2> 9395821 INFO  
(recoveryExecutor-3700-thread-1-processing-n:127.0.0.1:57668_solr 
x:recoverytest_shard1_replica_n3 s:shard1 c:recoverytest r:core_node4) 
[n:127.0.0.1:57668_solr c:recoverytest s:shard1 r:core_node4 
x:recoverytest_shard1_replica_n3] o.a.s.c.SolrCore 
[recoverytest_shard1_replica_n3]  CLOSING SolrCore 
org.apache.solr.core.SolrCore@40f1dba6
   [junit4]   2> 9395821 INFO  
(recoveryExecutor-3700-thread-1-processing-n:127.0.0.1:57668_solr 
x:recoverytest_shard1_replica_n3 s:shard1 c:recoverytest r:core_node4) 
[n:127.0.0.1:57668_solr c:recoverytest s:shard1 r:core_node4 
x:recoverytest_shard1_replica_n3] o.a.s.m.SolrMetricManager Closing metric 
reporters for registry=solr.core.recoverytest.shard1.replica_n3, tag=1089592230
   [junit4]   2> 9395821 INFO  
(recoveryExecutor-3700-thread-1-processing-n:127.0.0.1:57668_solr 
x:recoverytest_shard1_replica_n3 s:shard1 c:recoverytest r:core_node4) 
[n:127.0.0.1:57668_solr c:recoverytest s:shard1 r:core_node4 
x:recoverytest_shard1_replica_n3] o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@6113cb3a: rootName = 
solr_57668, domain = solr.core.recoverytest.shard1.replica_n3, service url = 
null, agent id = null] for registry solr.core.recoverytest.shard1.replica_n3 / 
com.codahale.metrics.MetricRegistry@1fc2f37a
   [junit4]   2> 9395828 INFO  
(recoveryExecutor-3700-thread-1-processing-n:127.0.0.1:57668_solr 
x:recoverytest_shard1_replica_n3 s:shard1 c:recoverytest r:core_node4) 
[n:127.0.0.1:57668_solr c:recoverytest s:shard1 r:core_node4 
x:recoverytest_shard1_replica_n3] o.a.s.m.SolrMetricManager Closing metric 
reporters for registry=solr.collection.recoverytest.shard1.leader, 
tag=1089592230
   [junit4]   2> 9395829 WARN  
(recoveryExecutor-3700-thread-1-processing-n:127.0.0.1:57668_solr 
x:recoverytest_shard1_replica_n3 s:shard1 c:recoverytest r:core_node4) 
[n:127.0.0.1:57668_solr c:recoverytest s:shard1 r:core_node4 
x:recoverytest_shard1_replica_n3] o.a.s.c.RecoveryStrategy Stopping recovery 
for core=[recoverytest_shard1_replica_n3] coreNodeName=[core_node4]
   [junit4]   2> 9395830 WARN  
(updateExecutor-3699-thread-2-processing-n:127.0.0.1:57668_solr) 
[n:127.0.0.1:57668_solr c:recoverytest s:shard1 r:core_node4 
x:recoverytest_shard1_replica_n3] o.a.s.u.DefaultSolrCoreState Skipping 
recovery because Solr is shutdown
   [junit4]   2> 9395834 WARN  
(zkCallback-3702-thread-5-processing-n:127.0.0.1:57668_solr) 
[n:127.0.0.1:57668_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, 
but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 9395834 INFO  (jetty-closer-3680-thread-2) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@2ad256f6{/solr,null,UNAVAILABLE}
   [junit4]   2> 9395840 ERROR 
(SUITE-RecoveryZkTest-seed#[CEDB9D92D513606A]-worker) [    ] 
o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper 
server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 9395840 INFO  
(SUITE-RecoveryZkTest-seed#[CEDB9D92D513606A]-worker) [    ] 
o.a.s.c.ZkTestServer connecting to 127.0.0.1:53106 53106
   [junit4]   2> 9395932 INFO  (Thread-13460) [    ] o.a.s.c.ZkTestServer 
connecting to 127.0.0.1:53106 53106
   [junit4]   2> 9395932 WARN  (Thread-13460) [    ] o.a.s.c.ZkTestServer Watch 
limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2>        4       /solr/aliases.json
   [junit4]   2>        4       /solr/clusterprops.json
   [junit4]   2>        3       /solr/security.json
   [junit4]   2>        3       /solr/configs/conf
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2>        28      /solr/collections/recoverytest/state.json
   [junit4]   2>        4       /solr/clusterstate.json
   [junit4]   2>        2       
/solr/collections/recoverytest/leader_elect/shard1/election/98824866184232966-core_node2-n_0000000000
   [junit4]   2>        2       /solr/autoscaling.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2>        4       /solr/collections
   [junit4]   2>        3       /solr/live_nodes
   [junit4]   2>        2       /solr/overseer/queue
   [junit4]   2>        2       /solr/autoscaling/events/.auto_add_replicas
   [junit4]   2>        2       /solr/overseer/collection-queue-work
   [junit4]   2> 
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_CEDB9D92D513606A-001
   [junit4]   2> Oct 14, 2017 2:25:30 AM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked 
thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): 
{rnd_b=PostingsFormat(name=Memory), a_t=Lucene50(blocksize=128), 
a_i=PostingsFormat(name=Memory), id=PostingsFormat(name=Memory)}, 
docValues:{_version_=DocValuesFormat(name=Direct)}, maxPointsInLeafNode=1961, 
maxMBSortInHeap=6.275634190549594, sim=RandomSimilarity(queryNorm=false): {}, 
locale=he, timezone=America/Belem
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 
1.8.0_144 (64-bit)/cpus=4,threads=1,free=116861792,total=524812288
   [junit4]   2> NOTE: All tests run in this JVM: [TestSurroundQueryParser, 
ChaosMonkeyNothingIsSafeWithPullReplicasTest, OverseerTest, 
AssignBackwardCompatibilityTest, ComputePlanActionTest, 
FullSolrCloudDistribCmdsTest, TestCloudSchemaless, NotRequiredUniqueKeyTest, 
DistributedFacetPivotLargeTest, ConnectionManagerTest, HttpTriggerListenerTest, 
ReplaceNodeTest, ParsingFieldUpdateProcessorsTest, 
TestDelegationWithHadoopAuth, TestIndexSearcher, ShowFileRequestHandlerTest, 
UniqFieldsUpdateProcessorFactoryTest, ConfigureRecoveryStrategyTest, 
TermVectorComponentDistributedTest, TestQuerySenderListener, 
AnalyticsQueryTest, ImplicitSnitchTest, TestReversedWildcardFilterFactory, 
JvmMetricsTest, MultiThreadedOCPTest, V2StandaloneTest, 
TestDynamicFieldResource, TestSortByMinMaxFunction, BasicDistributedZk2Test, 
SolrJmxReporterTest, BadIndexSchemaTest, TestTolerantSearch, 
MetricsHandlerTest, TestLockTree, AddSchemaFieldsUpdateProcessorFactoryTest, 
CleanupOldIndexTest, TemplateUpdateProcessorTest, 
TestHighFrequencyDictionaryFactory, TestSQLHandler, TestSearcherReuse, 
TestClusterStateMutator, HLLUtilTest, TestQueryUtils, TestSolrConfigHandler, 
TestZkChroot, TestCollectionAPIs, BasicAuthStandaloneTest, 
TestSolrCloudWithKerberosAlt, TestReloadAndDeleteDocs, MultiTermTest, 
TermsComponentTest, TestFoldingMultitermQuery, TestPointFields, 
PingRequestHandlerTest, DirectUpdateHandlerTest, InfixSuggestersTest, 
ClassificationUpdateProcessorFactoryTest, NumericFieldsTest, 
TestFilteredDocIdSet, TestInPlaceUpdatesStandalone, DateMathParserTest, 
TestStressReorder, DistributedDebugComponentTest, TestRetrieveFieldsOptimizer, 
BigEndianAscendingWordDeserializerTest, SearchHandlerTest, HardAutoCommitTest, 
ConnectionReuseTest, TestInfoStreamLogging, HighlighterMaxOffsetTest, 
TestPushWriter, TestStandardQParsers, DateRangeFieldTest, 
TestConfigSetsAPIExclusivity, PeerSyncReplicationTest, SolrTestCaseJ4Test, 
ManagedSchemaRoundRobinCloudTest, SolrCoreCheckLockOnStartupTest, 
BlockJoinFacetRandomTest, TriLevelCompositeIdRoutingTest, TestCloudRecovery, 
TestDocTermOrds, TestReload, TestComplexPhraseQParserPlugin, 
XsltUpdateRequestHandlerTest, TestZkAclsWithHadoopAuth, TestJsonFacets, 
UUIDUpdateProcessorFallbackTest, TestSort, StatsComponentTest, 
TestExactStatsCache, LeaderElectionTest, TestRandomFlRTGCloud, 
TestExceedMaxTermLength, OverseerRolesTest, SimplePostToolTest, 
HdfsChaosMonkeySafeLeaderTest, TestTestInjection, MinimalSchemaTest, 
HdfsUnloadDistributedZkTest, TestLegacyField, HdfsRecoverLeaseTest, 
RecoveryZkTest]
   [junit4] Completed [724/744 (5!)] on J2 in 143.87s, 1 test, 1 error <<< 
FAILURES!

[...truncated 47003 lines...]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to