Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/22748/
Java: 64bit/jdk-10 -XX:-UseCompressedOops -XX:+UseParallelGC

1 tests failed.
FAILED:  
org.apache.solr.cloud.autoscaling.sim.TestComputePlanAction.testNodeAdded

Error Message:
ComputePlanAction should have computed exactly 1 operation, but was: 
[org.apache.solr.client.solrj.request.CollectionAdminRequest$MoveReplica@23657e96,
 
org.apache.solr.client.solrj.request.CollectionAdminRequest$MoveReplica@ed7985f]
 expected:<1> but was:<2>

Stack Trace:
java.lang.AssertionError: ComputePlanAction should have computed exactly 1 
operation, but was: 
[org.apache.solr.client.solrj.request.CollectionAdminRequest$MoveReplica@23657e96,
 
org.apache.solr.client.solrj.request.CollectionAdminRequest$MoveReplica@ed7985f]
 expected:<1> but was:<2>
        at 
__randomizedtesting.SeedInfo.seed([B196C5F89CDD2326:D455938F3E7E8B25]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at 
org.apache.solr.cloud.autoscaling.sim.TestComputePlanAction.testNodeAdded(TestComputePlanAction.java:314)
        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)




Build Log:
[...truncated 12910 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestComputePlanAction
   [junit4]   2> Creating dataDir: 
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.sim.TestComputePlanAction_B196C5F89CDD2326-001/init-core-data-001
   [junit4]   2> 341978 WARN  
(SUITE-TestComputePlanAction-seed#[B196C5F89CDD2326]-worker) [    ] 
o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=4 numCloses=4
   [junit4]   2> 341978 INFO  
(SUITE-TestComputePlanAction-seed#[B196C5F89CDD2326]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) 
w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 341979 INFO  
(SUITE-TestComputePlanAction-seed#[B196C5F89CDD2326]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: 
@org.apache.solr.util.RandomizeSSL(reason="", ssl=0.0/0.0, value=0.0/0.0, 
clientAuth=0.0/0.0)
   [junit4]   2> 341979 INFO  
(SUITE-TestComputePlanAction-seed#[B196C5F89CDD2326]-worker) [    ] 
o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: 
test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 341995 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and 
.scheduled_maintenance triggers
   [junit4]   2> 341996 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 0
   [junit4]   2> 341996 DEBUG 
(SUITE-TestComputePlanAction-seed#[B196C5F89CDD2326]-worker) [    ] 
o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader: 
127.0.0.1:10000_solr
   [junit4]   2> 341996 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion -1
   [junit4]   2> 341996 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 0
   [junit4]   2> 342000 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial 
livenodes: [127.0.0.1:10000_solr]
   [junit4]   2> 342001 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 342001 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion 0
   [junit4]   2> 342001 DEBUG (ScheduledTrigger-3016-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 1
   [junit4]   2> 342003 INFO  
(SUITE-TestComputePlanAction-seed#[B196C5F89CDD2326]-worker) [    ] 
o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history 
in memory.
   [junit4]   2> 342004 INFO  
(TEST-TestComputePlanAction.testNodeLost-seed#[B196C5F89CDD2326]) [    ] 
o.a.s.SolrTestCaseJ4 ###Starting testNodeLost
   [junit4]   2> 342004 DEBUG 
(TEST-TestComputePlanAction.testNodeLost-seed#[B196C5F89CDD2326]) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
1
   [junit4]   2> 342004 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 342005 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 342005 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 342005 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 342005 INFO  
(TEST-TestComputePlanAction.testNodeLost-seed#[B196C5F89CDD2326]) [    ] 
o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing 
object cache...
   [junit4]   2> 342005 WARN  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Interrupted
   [junit4]   2> java.lang.InterruptedException: null
   [junit4]   2>        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2050)
 ~[?:?]
   [junit4]   2>        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2084)
 ~[?:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:189)
 [java/:?]
   [junit4]   2>        at java.lang.Thread.run(Thread.java:844) [?:?]
   [junit4]   2> 342005 DEBUG 
(TEST-TestComputePlanAction.testNodeLost-seed#[B196C5F89CDD2326]) [    ] 
o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 342006 DEBUG 
(TEST-TestComputePlanAction.testNodeLost-seed#[B196C5F89CDD2326]) [    ] 
o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 342006 DEBUG 
(TEST-TestComputePlanAction.testNodeLost-seed#[B196C5F89CDD2326]) [    ] 
o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 342006 DEBUG 
(TEST-TestComputePlanAction.testNodeLost-seed#[B196C5F89CDD2326]) [    ] 
o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool 
executor
   [junit4]   2> 342006 DEBUG 
(TEST-TestComputePlanAction.testNodeLost-seed#[B196C5F89CDD2326]) [    ] 
o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 342006 DEBUG 
(TEST-TestComputePlanAction.testNodeLost-seed#[B196C5F89CDD2326]) [    ] 
o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 342006 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and 
.scheduled_maintenance triggers
   [junit4]   2> 342006 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 2
   [junit4]   2> 342006 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion -1
   [junit4]   2> 342006 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 342006 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial 
livenodes: [127.0.0.1:10000_solr]
   [junit4]   2> 342007 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 342007 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
   [junit4]   2> 342007 DEBUG (ScheduledTrigger-3021-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 1
   [junit4]   2> 342029 DEBUG (ScheduledTrigger-3021-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 1
   [junit4]   2> 342050 DEBUG (ScheduledTrigger-3021-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 1
   [junit4]   2> 342070 DEBUG (ScheduledTrigger-3021-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 1
   [junit4]   2> 342090 DEBUG (ScheduledTrigger-3021-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 1
   [junit4]   2> 342114 DEBUG (ScheduledTrigger-3021-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 1
   [junit4]   2> 342123 DEBUG (simCloudManagerPool-3020-thread-1) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 342124 DEBUG (simCloudManagerPool-3020-thread-1) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
3
   [junit4]   2> 342124 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 342124 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 342124 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 342128 DEBUG (simCloudManagerPool-3020-thread-2) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 342130 DEBUG (simCloudManagerPool-3020-thread-2) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
4
   [junit4]   2> 342130 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 342131 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 342131 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
   [junit4]   2> 342135 DEBUG (ScheduledTrigger-3021-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 1
   [junit4]   2> 342155 DEBUG (ScheduledTrigger-3021-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 1
   [junit4]   2> 342176 DEBUG (ScheduledTrigger-3021-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 1
   [junit4]   2> 342196 DEBUG (ScheduledTrigger-3021-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 1
   [junit4]   2> 342216 DEBUG (ScheduledTrigger-3021-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 1
   [junit4]   2> 342236 DEBUG (ScheduledTrigger-3021-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 342237 DEBUG (simCloudManagerPool-3020-thread-3) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 342237 DEBUG (simCloudManagerPool-3020-thread-3) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
5
   [junit4]   2> 342238 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
   [junit4]   2> 342238 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeLostTrigger NodeLostTrigger node_lost_trigger - Initial 
livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
   [junit4]   2> 342238 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 342238 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
   [junit4]   2> 342238 DEBUG (ScheduledTrigger-3021-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 2
   [junit4]   2> 342238 DEBUG (simCloudManagerPool-3020-thread-4) [    ] 
o.a.s.c.o.ClusterStateMutator building a new cName: testNodeLost
   [junit4]   2> 342252 DEBUG (simCloudManagerPool-3020-thread-4) [    ] 
o.a.s.c.s.c.a.PolicyHelper Creating a new session
   [junit4]   2> 342253 DEBUG (simCloudManagerPool-3020-thread-4) [    ] 
o.a.s.c.s.c.a.PolicyHelper New session created 
   [junit4]   2> 342256 DEBUG (ScheduledTrigger-3021-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 342258 DEBUG (ScheduledTrigger-3021-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 2
   [junit4]   2> 342267 INFO  (simCloudManagerPool-3020-thread-4) [    ] 
o.a.s.c.s.c.a.PolicyHelper returnSession, curr-time 898811643 
sessionWrapper.createTime 898810937876308, this.sessionWrapper.createTime 
898810937876308 
   [junit4]   2> 342267 DEBUG (simCloudManagerPool-3020-thread-4) [    ] 
o.a.s.c.s.c.a.PolicyHelper session set to NULL
   [junit4]   2> 342277 DEBUG (ScheduledTrigger-3021-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 342278 DEBUG (ScheduledTrigger-3021-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 2
   [junit4]   2> 342297 DEBUG (ScheduledTrigger-3021-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 342299 DEBUG (ScheduledTrigger-3021-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 2
   [junit4]   2> 342302 DEBUG (simCloudManagerPool-3020-thread-5) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- submit leader election for testNodeLost 
/ shard1
   [junit4]   2> 342308 DEBUG (simCloudManagerPool-3020-thread-7) [    ] 
o.a.s.c.a.s.SimClusterStateProvider Running leader election for testNodeLost / 
shard1
   [junit4]   2> 342309 DEBUG (simCloudManagerPool-3020-thread-7) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for testNodeLost / 
shard1: core_node1
   [junit4]   2> 342317 DEBUG (ScheduledTrigger-3021-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 342319 DEBUG (ScheduledTrigger-3021-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 2
   [junit4]   2> 342319 DEBUG (simCloudManagerPool-3020-thread-6) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- already has leader testNodeLost / shard1
   [junit4]   2> 342319 DEBUG 
(TEST-TestComputePlanAction.testNodeLost-seed#[B196C5F89CDD2326]) [    ] 
o.a.s.c.a.s.TestComputePlanAction -- cluster state: znodeVersion: 6
   [junit4]   2> live nodes:[127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
   [junit4]   2> 
collections:{testNodeLost=DocCollection(testNodeLost//clusterstate.json/5)={
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "pullReplicas":"0",
   [junit4]   2>   "router":{"name":"compositeId"},
   [junit4]   2>   "maxShardsPerNode":"1",
   [junit4]   2>   "autoAddReplicas":"false",
   [junit4]   2>   "nrtReplicas":"2",
   [junit4]   2>   "tlogReplicas":"0",
   [junit4]   2>   "autoCreated":"true",
   [junit4]   2>   "shards":{"shard1":{
   [junit4]   2>       "replicas":{
   [junit4]   2>         "core_node1":{
   [junit4]   2>           "core":"testNodeLost_shard1_replica_n1",
   [junit4]   2>           "leader":"true",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "node_name":"127.0.0.1:10001_solr",
   [junit4]   2>           "state":"active",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0},
   [junit4]   2>         "core_node2":{
   [junit4]   2>           "core":"testNodeLost_shard1_replica_n2",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "node_name":"127.0.0.1:10000_solr",
   [junit4]   2>           "state":"active",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       "range":"80000000-7fffffff",
   [junit4]   2>       "state":"active"}}}}
   [junit4]   2> 342321 DEBUG 
(TEST-TestComputePlanAction.testNodeLost-seed#[B196C5F89CDD2326]) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- submit leader election for testNodeLost 
/ shard1
   [junit4]   2> 342321 INFO  
(TEST-TestComputePlanAction.testNodeLost-seed#[B196C5F89CDD2326]) [    ] 
o.a.s.c.a.s.TestComputePlanAction Stopped_node : 127.0.0.1:10001_solr
   [junit4]   2> 342321 DEBUG (simCloudManagerPool-3020-thread-8) [    ] 
o.a.s.c.a.s.SimClusterStateProvider Running leader election for testNodeLost / 
shard1
   [junit4]   2> 342321 INFO  (simCloudManagerPool-3020-thread-8) [    ] 
o.a.s.c.ActionThrottle Throttling leader attempts - waiting for 4418ms
   [junit4]   2> 342337 DEBUG (ScheduledTrigger-3021-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 342337 DEBUG (ScheduledTrigger-3021-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10001_solr
   [junit4]   2> 342339 DEBUG (ScheduledTrigger-3021-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 2
   [junit4]   2> 342339 DEBUG (ScheduledTrigger-3021-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10001_solr
   [junit4]   2> 342358 DEBUG (ScheduledTrigger-3021-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 342359 DEBUG (ScheduledTrigger-3021-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 2
   [junit4]   2> 342378 DEBUG (ScheduledTrigger-3021-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 342380 DEBUG (ScheduledTrigger-3021-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 2
   [junit4]   2> 342398 DEBUG (ScheduledTrigger-3021-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 342400 DEBUG (ScheduledTrigger-3021-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 2
   [junit4]   2> 342409 DEBUG (simCloudManagerPool-3020-thread-8) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for testNodeLost / 
shard1: core_node2
   [junit4]   2> 342419 DEBUG (ScheduledTrigger-3021-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 342420 DEBUG (ScheduledTrigger-3021-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 2
   [junit4]   2> 342439 DEBUG (ScheduledTrigger-3021-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 342440 DEBUG (ScheduledTrigger-3021-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 2
   [junit4]   2> 342459 DEBUG (ScheduledTrigger-3021-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 342460 DEBUG (ScheduledTrigger-3021-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 2
   [junit4]   2> 342479 DEBUG (ScheduledTrigger-3021-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 342481 DEBUG (ScheduledTrigger-3021-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 2
   [junit4]   2> 342481 DEBUG (ScheduledTrigger-3021-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost 
nodes: [127.0.0.1:10001_solr]
   [junit4]   2> 342482 DEBUG (ScheduledTrigger-3021-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"33177bb830a4aTagfrec5zzlm65qw4au9xvp1fm",
   [junit4]   2>   "source":"node_lost_trigger",
   [junit4]   2>   "eventTime":898815246928458,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[898815246928458],
   [junit4]   2>     "nodeNames":["127.0.0.1:10001_solr"]}}
   [junit4]   2> 342483 DEBUG (ScheduledTrigger-3021-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, 
.scheduled_maintenance, node_lost_trigger]
   [junit4]   2> 342485 DEBUG (simCloudManagerPool-3020-thread-9) [    ] 
o.a.s.c.o.ClusterStateMutator building a new cName: .system
   [junit4]   2> 342496 DEBUG (simCloudManagerPool-3020-thread-9) [    ] 
o.a.s.c.s.c.a.PolicyHelper Creating a new session
   [junit4]   2> 342497 DEBUG (simCloudManagerPool-3020-thread-9) [    ] 
o.a.s.c.s.c.a.PolicyHelper New session created 
   [junit4]   2> 342498 INFO  (simCloudManagerPool-3020-thread-9) [    ] 
o.a.s.c.s.c.a.PolicyHelper returnSession, curr-time 898823186 
sessionWrapper.createTime 898823152086958, this.sessionWrapper.createTime 
898823152086958 
   [junit4]   2> 342498 DEBUG (simCloudManagerPool-3020-thread-9) [    ] 
o.a.s.c.s.c.a.PolicyHelper session set to NULL
   [junit4]   2> 342512 DEBUG (simCloudManagerPool-3020-thread-10) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- submit leader election for .system / 
shard1
   [junit4]   2> 342521 DEBUG (simCloudManagerPool-3020-thread-11) [    ] 
o.a.s.c.a.s.SimClusterStateProvider Running leader election for .system / shard1
   [junit4]   2> 342521 DEBUG (simCloudManagerPool-3020-thread-11) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for .system / shard1: 
core_node1
   [junit4]   2> 342528 DEBUG (AutoscalingActionExecutor-3022-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"33177bb830a4aTagfrec5zzlm65qw4au9xvp1fm",
   [junit4]   2>   "source":"node_lost_trigger",
   [junit4]   2>   "eventTime":898815246928458,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[898815246928458],
   [junit4]   2>     "_enqueue_time_":898822438122558,
   [junit4]   2>     "nodeNames":["127.0.0.1:10001_solr"]}}
   [junit4]   2> 342529 DEBUG (AutoscalingActionExecutor-3022-thread-1) [    ] 
o.a.s.c.a.ComputePlanAction -- processing event: {
   [junit4]   2>   "id":"33177bb830a4aTagfrec5zzlm65qw4au9xvp1fm",
   [junit4]   2>   "source":"node_lost_trigger",
   [junit4]   2>   "eventTime":898815246928458,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[898815246928458],
   [junit4]   2>     "_enqueue_time_":898822438122558,
   [junit4]   2>     "nodeNames":["127.0.0.1:10001_solr"]}} with context 
properties: {BEFORE_ACTION=[compute_plan]}
   [junit4]   2> 342529 DEBUG (AutoscalingActionExecutor-3022-thread-1) [    ] 
o.a.s.c.s.c.a.PolicyHelper Creating a new session
   [junit4]   2> 342547 DEBUG (AutoscalingActionExecutor-3022-thread-1) [    ] 
o.a.s.c.s.c.a.PolicyHelper New session created 
   [junit4]   2> 342551 INFO  (AutoscalingActionExecutor-3022-thread-1) [    ] 
o.a.s.c.a.ComputePlanAction Computed Plan: 
action=MOVEREPLICA&collection=testNodeLost&targetNode=127.0.0.1:10002_solr&inPlaceMove=true&replica=core_node1
   [junit4]   2> 342551 INFO  (AutoscalingActionExecutor-3022-thread-1) [    ] 
o.a.s.c.s.c.a.PolicyHelper returnSession, curr-time 898825851 
sessionWrapper.createTime 898825642457508, this.sessionWrapper.createTime 
898825642457508 
   [junit4]   2> 342551 DEBUG (AutoscalingActionExecutor-3022-thread-1) [    ] 
o.a.s.c.s.c.a.PolicyHelper session set to NULL
   [junit4]   2> 342557 DEBUG 
(TEST-TestComputePlanAction.testNodeLost-seed#[B196C5F89CDD2326]) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- submit leader election for .system / 
shard1
   [junit4]   2> 342564 INFO  
(TEST-TestComputePlanAction.testNodeLost-seed#[B196C5F89CDD2326]) [    ] 
o.a.s.c.a.s.TestComputePlanAction -------------_ FINAL STATE --------------
   [junit4]   2> 342564 INFO  
(TEST-TestComputePlanAction.testNodeLost-seed#[B196C5F89CDD2326]) [    ] 
o.a.s.c.a.s.TestComputePlanAction * Node values: {
   [junit4]   2>   "127.0.0.1:10001_solr":{
   [junit4]   2>     "sysprop.java.vendor":"Oracle Corporation",
   [junit4]   2>     "cores":1,
   [junit4]   2>     "host":"127.0.0.1",
   [junit4]   2>     "freedisk":1023,
   [junit4]   2>     "sysprop.java.version":"10",
   [junit4]   2>     "ip_1":"127",
   [junit4]   2>     "ip_3":"0",
   [junit4]   2>     "ip_2":"0",
   [junit4]   2>     "ip_4":"1",
   [junit4]   2>     "node":"127.0.0.1:10001_solr",
   [junit4]   2>     "metrics:solr.jvm:buffers.direct.Count":0,
   [junit4]   2>     "port":10001,
   [junit4]   2>     "heapUsage":123450000,
   [junit4]   2>     "sysLoadAvg":1.0,
   [junit4]   2>     
"metrics:solr.node:ADMIN./admin/authorization.clientErrors:count":0},
   [junit4]   2>   "127.0.0.1:10000_solr":{
   [junit4]   2>     "sysprop.java.vendor":"Oracle Corporation",
   [junit4]   2>     "cores":1,
   [junit4]   2>     "host":"127.0.0.1",
   [junit4]   2>     "freedisk":999,
   [junit4]   2>     "sysprop.java.version":"10",
   [junit4]   2>     "ip_1":"127",
   [junit4]   2>     "ip_3":"0",
   [junit4]   2>     "ip_2":"0",
   [junit4]   2>     "ip_4":"1",
   [junit4]   2>     "node":"127.0.0.1:10000_solr",
   [junit4]   2>     "metrics:solr.jvm:buffers.direct.Count":0,
   [junit4]   2>     "port":10000,
   [junit4]   2>     "heapUsage":123450000,
   [junit4]   2>     "sysLoadAvg":1.0,
   [junit4]   2>     
"metrics:solr.node:ADMIN./admin/authorization.clientErrors:count":0},
   [junit4]   2>   "127.0.0.1:10002_solr":{
   [junit4]   2>     "sysprop.java.vendor":"Oracle Corporation",
   [junit4]   2>     "cores":1,
   [junit4]   2>     "host":"127.0.0.1",
   [junit4]   2>     "freedisk":1023,
   [junit4]   2>     "sysprop.java.version":"10",
   [junit4]   2>     "ip_1":"127",
   [junit4]   2>     "ip_3":"0",
   [junit4]   2>     "ip_2":"0",
   [junit4]   2>     "ip_4":"1",
   [junit4]   2>     "node":"127.0.0.1:10002_solr",
   [junit4]   2>     "metrics:solr.jvm:buffers.direct.Count":0,
   [junit4]   2>     "port":10002,
   [junit4]   2>     "heapUsage":123450000,
   [junit4]   2>     "sysLoadAvg":1.0,
   [junit4]   2>     
"metrics:solr.node:ADMIN./admin/authorization.clientErrors:count":0}}
   [junit4]   2> 342564 INFO  
(TEST-TestComputePlanAction.testNodeLost-seed#[B196C5F89CDD2326]) [    ] 
o.a.s.c.a.s.TestComputePlanAction * Live nodes: [127.0.0.1:10000_solr]
   [junit4]   2> 342565 INFO  
(TEST-TestComputePlanAction.testNodeLost-seed#[B196C5F89CDD2326]) [    ] 
o.a.s.c.a.s.TestComputePlanAction * Collection .system state: 
DocCollection(.system//clusterstate.json/13)={
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "pullReplicas":"0",
   [junit4]   2>   "router":{"name":"compositeId"},
   [junit4]   2>   "maxShardsPerNode":"1",
   [junit4]   2>   "autoAddReplicas":"false",
   [junit4]   2>   "nrtReplicas":"1",
   [junit4]   2>   "tlogReplicas":"0",
   [junit4]   2>   "autoCreated":"true",
   [junit4]   2>   "shards":{"shard1":{
   [junit4]   2>       "replicas":{"core_node1":{
   [junit4]   2>           "core":".system_shard1_replica_n1",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":2,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":11240,
   [junit4]   2>           "node_name":"127.0.0.1:10002_solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "INDEX.sizeInGB":1.0468065738677979E-5,
   [junit4]   2>           "SEARCHER.searcher.numDocs":2}},
   [junit4]   2>       "range":"80000000-7fffffff",
   [junit4]   2>       "state":"active"}}}
   [junit4]   2> 342565 INFO  
(TEST-TestComputePlanAction.testNodeLost-seed#[B196C5F89CDD2326]) [    ] 
o.a.s.c.a.s.TestComputePlanAction * Collection testNodeLost state: 
DocCollection(testNodeLost//clusterstate.json/13)={
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "pullReplicas":"0",
   [junit4]   2>   "router":{"name":"compositeId"},
   [junit4]   2>   "maxShardsPerNode":"1",
   [junit4]   2>   "autoAddReplicas":"false",
   [junit4]   2>   "nrtReplicas":"2",
   [junit4]   2>   "tlogReplicas":"0",
   [junit4]   2>   "autoCreated":"true",
   [junit4]   2>   "shards":{"shard1":{
   [junit4]   2>       "replicas":{
   [junit4]   2>         "core_node1":{
   [junit4]   2>           "core":"testNodeLost_shard1_replica_n1",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "node_name":"127.0.0.1:10001_solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0},
   [junit4]   2>         "core_node2":{
   [junit4]   2>           "core":"testNodeLost_shard1_replica_n2",
   [junit4]   2>           "leader":"true",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "node_name":"127.0.0.1:10000_solr",
   [junit4]   2>           "state":"active",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       "range":"80000000-7fffffff",
   [junit4]   2>       "state":"active"}}}
   [junit4]   2> 342565 DEBUG (simCloudManagerPool-3020-thread-13) [    ] 
o.a.s.c.a.s.SimClusterStateProvider Running leader election for .system / shard1
   [junit4]   2> 342565 INFO  
(TEST-TestComputePlanAction.testNodeLost-seed#[B196C5F89CDD2326]) [    ] 
o.a.s.SolrTestCaseJ4 ###Ending testNodeLost
   [junit4]   2> 342568 INFO  
(TEST-TestComputePlanAction.testNodeLost-seed#[B196C5F89CDD2326]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase #######################################
   [junit4]   2> ############ CLUSTER STATE ############
   [junit4]   2> #######################################
   [junit4]   2> ## Live nodes:         1
   [junit4]   2> ## Empty nodes:        0
   [junit4]   2> ## Dead nodes:         2
   [junit4]   2> ##             127.0.0.1:10001_solr
   [junit4]   2> ##             127.0.0.1:10002_solr
   [junit4]   2> ## Collections:        [.system, testNodeLost]
   [junit4]   2> ## Max replicas per node:      1
   [junit4]   2> ## Min replicas per node:      1
   [junit4]   2> ## Total replicas:             1
   [junit4]   2> ## * testNodeLost              1
   [junit4]   2> ##             - active           1
   [junit4]   2> ######### Solr op counts ##########
   [junit4]   2> ##             - CREATE             1
   [junit4]   2> ##             - autoscaling        3
   [junit4]   2> ##             - update             2
   [junit4]   2> ######### Autoscaling event counts ###########
   [junit4]   2> ## * Trigger: node_lost_trigger
   [junit4]   2> ##             - BEFORE_ACTION     1
   [junit4]   2> ##             - STARTED         1
   [junit4]   2> 
   [junit4]   2> 342584 WARN  (simCloudManagerPool-3020-thread-13) [    ] 
o.a.s.c.a.s.SimClusterStateProvider Can't find any active replicas for .system 
/ shard1: shard1:{
   [junit4]   2>   "replicas":{"core_node1":{
   [junit4]   2>       "core":".system_shard1_replica_n1",
   [junit4]   2>       "SEARCHER.searcher.maxDoc":2,
   [junit4]   2>       "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>       "INDEX.sizeInBytes":11240,
   [junit4]   2>       "node_name":"127.0.0.1:10002_solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "type":"NRT",
   [junit4]   2>       "INDEX.sizeInGB":1.0468065738677979E-5,
   [junit4]   2>       "SEARCHER.searcher.numDocs":2}},
   [junit4]   2>   "range":"80000000-7fffffff",
   [junit4]   2>   "state":"active"}
   [junit4]   2> 342585 DEBUG (simCloudManagerPool-3020-thread-13) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- liveNodes: [127.0.0.1:10000_solr]
   [junit4]   2> 342585 DEBUG (simCloudManagerPool-3020-thread-14) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- no leader in shard1:{
   [junit4]   2>   "replicas":{"core_node1":{
   [junit4]   2>       "core":".system_shard1_replica_n1",
   [junit4]   2>       "SEARCHER.searcher.maxDoc":2,
   [junit4]   2>       "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>       "INDEX.sizeInBytes":11240,
   [junit4]   2>       "node_name":"127.0.0.1:10002_solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "type":"NRT",
   [junit4]   2>       "INDEX.sizeInGB":1.0468065738677979E-5,
   [junit4]   2>       "SEARCHER.searcher.numDocs":2}},
   [junit4]   2>   "range":"80000000-7fffffff",
   [junit4]   2>   "state":"active"}
   [junit4]   2> 342602 INFO  
(TEST-TestComputePlanAction.testNodeWithMultipleReplicasLost-seed#[B196C5F89CDD2326])
 [    ] o.a.s.SolrTestCaseJ4 ###Starting testNodeWithMultipleReplicasLost
   [junit4]   2> 342602 DEBUG 
(TEST-TestComputePlanAction.testNodeWithMultipleReplicasLost-seed#[B196C5F89CDD2326])
 [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 6
   [junit4]   2> 342602 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 6
   [junit4]   2> 342602 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 342602 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread   -- deleted 
/autoscaling/nodeLost/127.0.0.1:10002_solr
   [junit4]   2> 342602 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 342602 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion 6
   [junit4]   2> 342603 DEBUG (simCloudManagerPool-3020-thread-15) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- no leader in shard1:{
   [junit4]   2>   "replicas":{"core_node1":{
   [junit4]   2>       "core":".system_shard1_replica_n1",
   [junit4]   2>       "SEARCHER.searcher.maxDoc":2,
   [junit4]   2>       "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>       "INDEX.sizeInBytes":11240,
   [junit4]   2>       "node_name":"127.0.0.1:10002_solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "type":"NRT",
   [junit4]   2>       "INDEX.sizeInGB":1.0468065738677979E-5,
   [junit4]   2>       "SEARCHER.searcher.numDocs":2}},
   [junit4]   2>   "range":"80000000-7fffffff",
   [junit4]   2>   "state":"active"}
   [junit4]   2> 342603 DEBUG 
(TEST-TestComputePlanAction.testNodeWithMultipleReplicasLost-seed#[B196C5F89CDD2326])
 [    ] o.a.s.c.a.s.SimNodeStateProvider - removing dead node values: 
127.0.0.1:10001_solr
   [junit4]   2> 342603 DEBUG 
(TEST-TestComputePlanAction.testNodeWithMultipleReplicasLost-seed#[B196C5F89CDD2326])
 [    ] o.a.s.c.a.s.SimNodeStateProvider - removing dead node values: 
127.0.0.1:10002_solr
   [junit4]   2> 342603 INFO  
(TEST-TestComputePlanAction.testNodeWithMultipleReplicasLost-seed#[B196C5F89CDD2326])
 [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and 
clearing object cache...
   [junit4]   2> 342603 DEBUG 
(TEST-TestComputePlanAction.testNodeWithMultipleReplicasLost-seed#[B196C5F89CDD2326])
 [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool 
executor now
   [junit4]   2> 342603 WARN  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are 
closed, exiting.
   [junit4]   2> 342603 DEBUG 
(TEST-TestComputePlanAction.testNodeWithMultipleReplicasLost-seed#[B196C5F89CDD2326])
 [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 342603 DEBUG 
(TEST-TestComputePlanAction.testNodeWithMultipleReplicasLost-seed#[B196C5F89CDD2326])
 [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 342604 DEBUG (AutoscalingActionExecutor-3022-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing took 3826 ms for event 
id=33177bb830a4aTagfrec5zzlm65qw4au9xvp1fm
   [junit4]   2> 342604 DEBUG 
(TEST-TestComputePlanAction.testNodeWithMultipleReplicasLost-seed#[B196C5F89CDD2326])
 [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread 
pool executor
   [junit4]   2> 342604 DEBUG 
(TEST-TestComputePlanAction.testNodeWithMultipleReplicasLost-seed#[B196C5F89CDD2326])
 [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 342604 DEBUG 
(TEST-TestComputePlanAction.testNodeWithMultipleReplicasLost-seed#[B196C5F89CDD2326])
 [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed 
explicitly
   [junit4]   2> 342609 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and 
.scheduled_maintenance triggers
   [junit4]   2> 342609 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 7
   [junit4]   2> 342609 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion -1
   [junit4]   2> 342609 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
   [junit4]   2> 342610 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial 
livenodes: [127.0.0.1:10000_solr]
   [junit4]   2> 342615 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 342615 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
   [junit4]   2> 342615 DEBUG (ScheduledTrigger-3024-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 1
   [junit4]   2> 342635 DEBUG (ScheduledTrigger-3024-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 1
   [junit4]   2> 342656 DEBUG (ScheduledTrigger-3024-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 1
   [junit4]   2> 342676 DEBUG (ScheduledTrigger-3024-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 1
   [junit4]   2> 342697 DEBUG (ScheduledTrigger-3024-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 1
   [junit4]   2> 342709 DEBUG (simCloudManagerPool-3023-thread-1) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 342710 DEBUG (simCloudManagerPool-3023-thread-1) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
8
   [junit4]   2> 342710 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 8
   [junit4]   2> 342710 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 342710 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 8, lastZnodeVersion 8
   [junit4]   2> 342711 DEBUG (simCloudManagerPool-3023-thread-2) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 342712 DEBUG (simCloudManagerPool-3023-thread-2) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
9
   [junit4]   2> 342712 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 9
   [junit4]   2> 342712 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 342712 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion 9
   [junit4]   2> 342718 DEBUG (ScheduledTrigger-3024-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 1
   [junit4]   2> 342738 DEBUG (ScheduledTrigger-3024-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 1
   [junit4]   2> 342759 DEBUG (ScheduledTrigger-3024-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 1
   [junit4]   2> 342779 DEBUG (ScheduledTrigger-3024-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 1
   [junit4]   2> 342799 DEBUG (ScheduledTrigger-3024-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 1
   [junit4]   2> 342814 DEBUG (simCloudManagerPool-3023-thread-3) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 342814 DEBUG (simCloudManagerPool-3023-thread-3) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
10
   [junit4]   2> 342814 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 10
   [junit4]   2> 342815 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeLostTrigger NodeLostTrigger node_lost_trigger - Initial 
livenodes: [127.0.0.1:10004_solr, 127.0.0.1:10005_solr, 127.0.0.1:10000_solr, 
127.0.0.1:10003_solr]
   [junit4]   2> 342815 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 342815 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 10, lastZnodeVersion 10
   [junit4]   2> 342815 DEBUG (ScheduledTrigger-3024-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 4
   [junit4]   2> 342817 DEBUG (simCloudManagerPool-3023-thread-4) [    ] 
o.a.s.c.o.ClusterStateMutator building a new cName: 
testNodeWithMultipleReplicasLost
   [junit4]   2> 342819 DEBUG (ScheduledTrigger-3024-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4
   [junit4]   2> 342829 DEBUG (simCloudManagerPool-3023-thread-4) [    ] 
o.a.s.c.s.c.a.PolicyHelper Creating a new session
   [junit4]   2> 342831 DEBUG (simCloudManagerPool-3023-thread-4) [    ] 
o.a.s.c.s.c.a.PolicyHelper New session created 
   [junit4]   2> 342836 DEBUG (ScheduledTrigger-3024-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 4
   [junit4]   2> 342840 DEBUG (ScheduledTrigger-3024-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4
   [junit4]   2> 342850 INFO  (simCloudManagerPool-3023-thread-4) [    ] 
o.a.s.c.s.c.a.PolicyHelper returnSession, curr-time 898840762 
sessionWrapper.createTime 898839849763808, this.sessionWrapper.createTime 
898839849763808 
   [junit4]   2> 342850 DEBUG (simCloudManagerPool-3023-thread-4) [    ] 
o.a.s.c.s.c.a.PolicyHelper session set to NULL
   [junit4]   2> 342856 DEBUG (ScheduledTrigger-3024-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 4
   [junit4]   2> 342860 DEBUG (ScheduledTrigger-3024-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4
   [junit4]   2> 342873 DEBUG (simCloudManagerPool-3023-thread-5) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- submit leader election for 
testNodeWithMultipleReplicasLost / shard1
   [junit4]   2> 342876 DEBUG (ScheduledTrigger-3024-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 4
   [junit4]   2> 342880 DEBUG (ScheduledTrigger-3024-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4
   [junit4]   2> 342881 DEBUG (simCloudManagerPool-3023-thread-9) [    ] 
o.a.s.c.a.s.SimClusterStateProvider Running leader election for 
testNodeWithMultipleReplicasLost / shard1
   [junit4]   2> 342896 DEBUG (ScheduledTrigger-3024-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 4
   [junit4]   2> 342900 DEBUG (ScheduledTrigger-3024-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4
   [junit4]   2> 342900 DEBUG (simCloudManagerPool-3023-thread-9) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for 
testNodeWithMultipleReplicasLost / shard1: core_node1
   [junit4]   2> 342904 DEBUG (simCloudManagerPool-3023-thread-6) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- no replicas in 
testNodeWithMultipleReplicasLost / shard2
   [junit4]   2> 342906 DEBUG (simCloudManagerPool-3023-thread-6) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- already has leader 
testNodeWithMultipleReplicasLost / shard1
   [junit4]   2> 342917 DEBUG (ScheduledTrigger-3024-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 4
   [junit4]   2> 342920 DEBUG (ScheduledTrigger-3024-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4
   [junit4]   2> 342922 DEBUG (simCloudManagerPool-3023-thread-7) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- no replicas in 
testNodeWithMultipleReplicasLost / shard2
   [junit4]   2> 342922 DEBUG (simCloudManagerPool-3023-thread-7) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- already has leader 
testNodeWithMultipleReplicasLost / shard1
   [junit4]   2> 342933 DEBUG (simCloudManagerPool-3023-thread-8) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- submit leader election for 
testNodeWithMultipleReplicasLost / shard2
   [junit4]   2> 342936 DEBUG (simCloudManagerPool-3023-thread-8) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- already has leader 
testNodeWithMultipleReplicasLost / shard1
   [junit4]   2> 342937 DEBUG (ScheduledTrigger-3024-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 4
   [junit4]   2> 342941 DEBUG (ScheduledTrigger-3024-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4
   [junit4]   2> 342985 DEBUG (ScheduledTrigger-3024-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 4
   [junit4]   2> 342985 DEBUG (ScheduledTrigger-3024-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4
   [junit4]   2> 342985 DEBUG (simCloudManagerPool-3023-thread-10) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- submit leader election for 
testNodeWithMultipleReplicasLost / shard2
   [junit4]   2> 342985 DEBUG (simCloudManagerPool-3023-thread-12) [    ] 
o.a.s.c.a.s.SimClusterStateProvider Running leader election for 
testNodeWithMultipleReplicasLost / shard2
   [junit4]   2> 342985 DEBUG (simCloudManagerPool-3023-thread-12) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for 
testNodeWithMultipleReplicasLost / shard2: core_node4
   [junit4]   2> 342992 DEBUG (simCloudManagerPool-3023-thread-10) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- already has leader 
testNodeWithMultipleReplicasLost / shard1
   [junit4]   2> 342992 DEBUG (simCloudManagerPool-3023-thread-13) [    ] 
o.a.s.c.a.s.SimClusterStateProvider Running leader election for 
testNodeWithMultipleReplicasLost / shard2
   [junit4]   2> 342993 DEBUG (simCloudManagerPool-3023-thread-13) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- already has leader 
testNodeWithMultipleReplicasLost / shard2: shard2:{
   [junit4]   2>   "replicas":{
   [junit4]   2>     "core_node4":{
   [junit4]   2>       
"core":"testNodeWithMultipleReplicasLost_shard2_replica_n4",
   [junit4]   2>       "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>       "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>       "INDEX.sizeInBytes":10240,
   [junit4]   2>       "node_name":"127.0.0.1:10000_solr",
   [junit4]   2>       "state":"active",
   [junit4]   2>       "type":"NRT",
   [junit4]   2>       "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>       "SEARCHER.searcher.numDocs":0},
   [junit4]   2>     "core_node5":{
   [junit4]   2>       
"core":"testNodeWithMultipleReplicasLost_shard2_replica_n5",
   [junit4]   2>       "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>       "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>       "INDEX.sizeInBytes":10240,
   [junit4]   2>       "node_name":"127.0.0.1:10004_solr",
   [junit4]   2>       "state":"active",
   [junit4]   2>       "type":"NRT",
   [junit4]   2>       "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>       "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>   "range":"0-7fffffff",
   [junit4]   2>   "state":"active"}
   [junit4]   2> 343003 DEBUG (simCloudManagerPool-3023-thread-11) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- already has leader 
testNodeWithMultipleReplicasLost / shard2
   [junit4]   2> 343003 DEBUG (simCloudManagerPool-3023-thread-11) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- already has leader 
testNodeWithMultipleReplicasLost / shard1
   [junit4]   2> 343003 DEBUG 
(TEST-TestComputePlanAction.testNodeWithMultipleReplicasLost-seed#[B196C5F89CDD2326])
 [    ] o.a.s.c.a.s.TestComputePlanAction -- cluster state: znodeVersion: 25
   [junit4]   2> live nodes:[127.0.0.1:10004_solr, 127.0.0.1:10000_solr, 
127.0.0.1:10003_solr, 127.0.0.1:10005_solr]
   [junit4]   2> 
collections:{testNodeWithMultipleReplicasLost=DocCollection(testNodeWithMultipleReplicasLost//clusterstate.json/24)={
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "pullReplicas":"0",
   [junit4]   2>   "router":{"name":"compositeId"},
   [junit4]   2>   "maxShardsPerNode":"1",
   [junit4]   2>   "autoAddReplicas":"false",
   [junit4]   2>   "nrtReplicas":"3",
   [junit4]   2>   "tlogReplicas":"0",
   [junit4]   2>   "autoCreated":"true",
   [junit4]   2>   "shards":{
   [junit4]   2>     "shard2":{
   [junit4]   2>       "replicas":{
   [junit4]   2>         "core_node6":{
   [junit4]   2>           
"core":"testNodeWithMultipleReplicasLost_shard2_replica_n6",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "node_name":"127.0.0.1:10005_solr",
   [junit4]   2>           "state":"active",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0},
   [junit4]   2>         "core_node4":{
   [junit4]   2>           
"core":"testNodeWithMultipleReplicasLost_shard2_replica_n4",
   [junit4]   2>           "leader":"true",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "node_name":"127.0.0.1:10000_solr",
   [junit4]   2>           "state":"active",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0},
   [junit4]   2>         "core_node5":{
   [junit4]   2>           
"core":"testNodeWithMultipleReplicasLost_shard2_replica_n5",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "node_name":"127.0.0.1:10004_solr",
   [junit4]   2>           "state":"active",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       "range":"0-7fffffff",
   [junit4]   2>       "state":"active"},
   [junit4]   2>     "shard1":{
   [junit4]   2>       "replicas":{
   [junit4]   2>         "core_node1":{
   [junit4]   2>           
"core":"testNodeWithMultipleReplicasLost_shard1_replica_n1",
   [junit4]   2>           "leader":"true",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "node_name":"127.0.0.1:10003_solr",
   [junit4]   2>           "state":"active",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0},
   [junit4]   2>         "core_node2":{
   [junit4]   2>           
"core":"testNodeWithMultipleReplicasLost_shard1_replica_n2",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "node_name":"127.0.0.1:10005_solr",
   [junit4]   2>           "state":"active",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0},
   [junit4]   2>         "core_node3":{
   [junit4]   2>           
"core":"testNodeWithMultipleReplicasLost_shard1_replica_n3",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "node_name":"127.0.0.1:10004_solr",
   [junit4]   2>           "state":"active",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       "range":"80000000-ffffffff",
   [junit4]   2>       "state":"active"}}}}
   [junit4]   2> 343004 DEBUG 
(TEST-TestComputePlanAction.testNodeWithMultipleReplicasLost-seed#[B196C5F89CDD2326])
 [    ] o.a.s.c.a.s.SimClusterStateProvider -- already has leader 
testNodeWithMultipleReplicasLost / shard2
   [junit4]   2> 343004 DEBUG 
(TEST-TestComputePlanAction.testNodeWithMultipleReplicasLost-seed#[B196C5F89CDD2326])
 [    ] o.a.s.c.a.s.SimClusterStateProvider -- already has leader 
testNodeWithMultipleReplicasLost / shard1
   [junit4]   2> 343005 DEBUG (ScheduledTrigger-3024-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 3
   [junit4]   2> 343005 DEBUG (ScheduledTrigger-3024-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10004_solr
   [junit4]   2> 343005 DEBUG (ScheduledTrigger-3024-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 343005 DEBUG (ScheduledTrigger-3024-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10004_solr
   [junit4]   2> 343025 DEBUG (ScheduledTrigger-3024-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 3
   [junit4]   2> 343025 DEBUG (ScheduledTrigger-3024-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost 
nodes: [127.0.0.1:10004_solr]
   [junit4]   2> 343025 DEBUG (ScheduledTrigger-3024-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"3317f7bdd15e0Tagfrec5zzlm65qw4au9xvp1fr",
   [junit4]   2>   "source":"node_lost_trigger",
   [junit4]   2>   "eventTime":898848538826208,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[898848538826208],
   [junit4]   2>     "nodeNames":["127.0.0.1:10004_solr"]}}
   [junit4]   2> 343026 DEBUG (ScheduledTrigger-3024-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, 
.scheduled_maintenance, node_lost_trigger]
   [junit4]   2> 343027 DEBUG (simCloudManagerPool-3023-thread-14) [    ] 
o.a.s.c.o.ClusterStateMutator building a new cName: .system
   [junit4]   2> 343039 DEBUG (simCloudManagerPool-3023-thread-14) [    ] 
o.a.s.c.s.c.a.PolicyHelper Creating a new session
   [junit4]   2> 343039 DEBUG (simCloudManagerPool-3023-thread-14) [    ] 
o.a.s.c.s.c.a.PolicyHelper New session created 
   [junit4]   2> 343040 INFO  (simCloudManagerPool-3023-thread-14) [    ] 
o.a.s.c.s.c.a.PolicyHelper returnSession, curr-time 898850277 
sessionWrapper.createTime 898850245257708, this.sessionWrapper.createTime 
898850245257708 
   [junit4]   2> 343040 DEBUG (simCloudManagerPool-3023-thread-14) [    ] 
o.a.s.c.s.c.a.PolicyHelper session set to NULL
   [junit4]   2> 343069 DEBUG (simCloudManagerPool-3023-thread-15) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- submit leader election for .system / 
shard1
   [junit4]   2> 343071 DEBUG (simCloudManagerPool-3023-thread-16) [    ] 
o.a.s.c.a.s.SimClusterStateProvider Running leader election for .system / shard1
   [junit4]   2> 343072 DEBUG (simCloudManagerPool-3023-thread-16) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for .system / shard1: 
core_node2
   [junit4]   2> 343083 DEBUG (AutoscalingActionExecutor-3025-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"3317f7bdd15e0Tagfrec5zzlm65qw4au9xvp1fr",
   [junit4]   2>   "source":"node_lost_trigger",
   [junit4]   2>   "eventTime":898848538826208,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[898848538826208],
   [junit4]   2>     "_enqueue_time_":898849567895008,
   [junit4]   2>     "nodeNames":["127.0.0.1:10004_solr"]}}
   [junit4]   2> 343083 DEBUG (AutoscalingActionExecutor-3025-thread-1) [    ] 
o.a.s.c.a.ComputePlanAction -- processing event: {
   [junit4]   2>   "id":"3317f7bdd15e0Tagfrec5zzlm65qw4au9xvp1fr",
   [junit4]   2>   "source":"node_lost_trigger",
   [junit4]   2>   "eventTime":898848538826208,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[898848538826208],
   [junit4]   2>     "_enqueue_time_":898849567895008,
   [junit4]   2>     "nodeNames":["127.0.0.1:10004_solr"]}} with context 
properties: {BEFORE_ACTION=[compute_plan]}
   [junit4]   2> 343083 DEBUG (AutoscalingActionExecutor-3025-thread-1) [    ] 
o.a.s.c.s.c.a.PolicyHelper Creating a new session
   [junit4]   2> 343084 DEBUG (AutoscalingActionExecutor-3025-thread-1) [    ] 
o.a.s.c.s.c.a.PolicyHelper New session created 
   [junit4]   2> 343087 INFO  (AutoscalingActionExecutor-3025-thread-1) [    ] 
o.a.s.c.a.ComputePlanAction Computed Plan: 
action=MOVEREPLICA&collection=testNodeWithMultipleReplicasLost&targetNode=127.0.0.1:10003_solr&inPlaceMove=true&replica=core_node5
   [junit4]   2> 343088 INFO  (AutoscalingActionExecutor-3025-thread-1) [    ] 
o.a.s.c.a.ComputePlanAction Computed Plan: 
action=MOVEREPLICA&collection=testNodeWithMultipleReplicasLost&targetNode=127.0.0.1:10000_solr&inPlaceMove=true&replica=core_node3
   [junit4]   2> 343088 INFO  (AutoscalingActionExecutor-3025-thread-1) [    ] 
o.a.s.c.s.c.a.PolicyHelper returnSession, curr-time 898852677 
sessionWrapper.createTime 898852494124408, this.sessionWrapper.createTime 
898852494124408 
   [junit4]   2> 343088 DEBUG (AutoscalingActionExecutor-3025-thread-1) [    ] 
o.a.s.c.s.c.a.PolicyHelper session set to NULL
   [junit4]   2> 343101 INFO  
(TEST-TestComputePlanAction.testNodeWithMultipleReplicasLost-seed#[B196C5F89CDD2326])
 [    ] o.a.s.c.a.s.TestComputePlanAction 
action=MOVEREPLICA&collection=testNodeWithMultipleReplicasLost&targetNode=127.0.0.1:10003_solr&inPlaceMove=true&replica=core_node5
   [junit4]   2> 343101 INFO  
(TEST-TestComputePlanAction.testNodeWithMultipleReplicasLost-seed#[B196C5F89CDD2326])
 [    ] o.a.s.c.a.s.TestComputePlanAction 
action=MOVEREPLICA&collection=testNodeWithMultipleReplicasLost&targetNode=127.0.0.1:10000_solr&inPlaceMove=true&replica=core_node3
   [junit4]   2> 343101 INFO  
(TEST-TestComputePlanAction.testNodeWithMultipleReplicasLost-seed#[B196C5F89CDD2326])
 [    ] o.a.s.c.a.s.TestComputePlanAction -------------_ FINAL STATE 
--------------
   [junit4]   2> 343101 INFO  
(TEST-TestComputePlanAction.testNodeWithMultipleReplicasLost-seed#[B196C5F89CDD2326])
 [    ] o.a.s.c.a.s.TestComputePlanAction * Node values: {
   [junit4]   2>   "127.0.0.1:10004_solr":{
   [junit4]   2>     "sysprop.java.vendor":"Oracle Corporation",
   [junit4]   2>     "cores":2,
   [junit4]   2>     "host":"127.0.0.1",
   [junit4]   2>     "freedisk":1022,
   [junit4]   2>     "sysprop.java.version":"10",
   [junit4]   2>     "ip_1":"127",
   [junit4]   2>     "ip_3":"0",
   [junit4]   2>     "ip_2":"0",
   [junit4]   2>     "ip_4":"1",
   [junit4]   2>     "node":"127.0.0.1:10004_solr",
   [junit4]   2>     "metrics:solr.jvm:buffers.direct.Count":0,
   [junit4]   2>     "port":10004,
   [junit4]   2>     "heapUsage":123450000,
   [junit4]   2>     "sysLoadAvg":1.0,
   [junit4]   2>     
"metrics:solr.node:ADMIN./admin/authorization.clientErrors:count":0},
   [junit4]   2>   "127.0.0.1:10005_solr":{
   [junit4]   2>     "sysprop.java.vendor":"Oracle Corporation",
   [junit4]   2>     "cores":2,
   [junit4]   2>     "host":"127.0.0.1",
   [junit4]   2>     "freedisk":1022,
   [junit4]   2>     "sysprop.java.version":"10",
   [junit4]   2>     "ip_1":"127",
   [junit4]   2>     "ip_3":"0",
   [junit4]   2>     "ip_2":"0",
   [junit4]   2>     "ip_4":"1",
   [junit4]   2>     "node":"127.0.0.1:10005_solr",
   [junit4]   2>     "metrics:solr.jvm:buffers.direct.Count":0,
   [junit4]   2>     "port":10005,
   [junit4]   2>     "heapUsage":123450000,
   [junit4]   2>     "sysLoadAvg":1.0,
   [junit4]   2>     
"metrics:solr.node:ADMIN./admin/authorization.clientErrors:count":0},
   [junit4]   2>   "127.0.0.1:10000_solr":{
   [junit4]   2>     "sysprop.java.vendor":"Oracle Corporation",
   [junit4]   2>     "cores":1,
   [junit4]   2>     "host":"127.0.0.1",
   [junit4]   2>     "freedisk":999,
   [junit4]   2>     "sysprop.java.version":"10",
   [junit4]   2>     "ip_1":"127",
   [junit4]   2>     "ip_3":"0",
   [junit4]   2>     "ip_2":"0",
   [junit4]   2>     "ip_4":"1",
   [junit4]   2>     "node":"127.0.0.1:10000_solr",
   [junit4]   2>     "metrics:solr.jvm:buffers.direct.Count":0,
   [junit4]   2>     "port":10000,
   [junit4]   2>     "heapUsage":123450000,
   [junit4]   2>     "sysLoadAvg":1.0,
   [junit4]   2>     
"metrics:solr.node:ADMIN./admin/authorization.clientErrors:count":0},
   [junit4]   2>   "127.0.0.1:10003_solr":{
   [junit4]   2>     "sysprop.java.vendor":"Oracle Corporation",
   [junit4]   2>     "cores":2,
   [junit4]   2>     "host":"127.0.0.1",
   [junit4]   2>     "freedisk":1022,
   [junit4]   2>     "sysprop.java.version":"10",
   [junit4]   2>     "ip_1":"127",
   [junit4]   2>     "ip_3":"0",
   [junit4]   2>     "ip_2":"0",
   [junit4]   2>     "ip_4":"1",
   [junit4]   2>     "node":"127.0.0.1:10003_solr",
   [junit4]   2>     "metrics:solr.jvm:buffers.direct.Count":0,
   [junit4]   2>     "port":10003,
   [junit4]   2>     "heapUsage":123450000,
   [junit4]   2>     "sysLoadAvg":1.0,
   [junit4]   2>     
"metrics:solr.node:ADMIN./admin/authorization.clientErrors:count":0}}
   [junit4]   2> 343101 INFO  
(TEST-TestComputePlanAction.testNodeWithMultipleReplicasLost-seed#[B196C5F89CDD2326])
 [    ] o.a.s.c.a.s.TestComputePlanAction * Live nodes: [127.0.0.1:10005_solr, 
127.0.0.1:10000_solr, 127.0.0.1:10003_solr]
   [junit4]   2> 343102 INFO  
(TEST-TestComputePlanAction.testNodeWithMultipleReplicasLost-seed#[B196C5F89CDD2326])
 [    ] o.a.s.c.a.s.TestComputePlanAction * Collection .system state: 
DocCollection(.system//clusterstate.json/32)={
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "pullReplicas":"0",
   [junit4]   2>   "router":{"name":"compositeId"},
   [junit4]   2>   "maxShardsPerNode":"1",
   [junit4]   2>   "autoAddReplicas":"false",
   [junit4]   2>   "nrtReplicas":"1",
   [junit4]   2>   "tlogReplicas":"0",
   [junit4]   2>   "autoCreated":"true",
   [junit4]   2>   "shards":{"shard1":{
   [junit4]   2>       "replicas":{"core_node2":{
   [junit4]   2>           "core":".system_shard1_replica_n1",
   [junit4]   2>           "leader":"true",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":2,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":11240,
   [junit4]   2>           "node_name":"127.0.0.1:10003_solr",
   [junit4]   2>           "state":"active",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "INDEX.sizeInGB":1.0468065738677979E-5,
   [junit4]   2>           "SEARCHER.searcher.numDocs":2}},
   [junit4]   2>       "range":"80000000-7fffffff",
   [junit4]   2>       "state":"active"}}}
   [junit4]   2> 343102 INFO  
(TEST-TestComputePlanAction.testNodeWithMultipleReplicasLost-seed#[B196C5F89CDD2326])
 [    ] o.a.s.c.a.s.TestComputePlanAction * Collection 
testNodeWithMultipleReplicasLost state: 
DocCollection(testNodeWithMultipleReplicasLost//clusterstate.json/32)={
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "pullReplicas":"0",
   [junit4]   2>   "router":{"name":"compositeId"},
   [junit4]   2>   "maxShardsPerNode":"1",
   [junit4]   2>   "autoAddReplicas":"false",
   [junit4]   2>   "nrtReplicas":"3",
   [junit4]   2>   "tlogReplicas":"0",
   [junit4]   2>   "autoCreated":"true",
   [junit4]   2>   "shards":{
   [junit4]   2>     "shard2":{
   [junit4]   2>       "replicas":{
   [junit4]   2>         "core_node6":{
   [junit4]   2>           
"core":"testNodeWithMultipleReplicasLost_shard2_replica_n6",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "node_name":"127.0.0.1:10005_solr",
   [junit4]   2>           "state":"active",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0},
   [junit4]   2>         "core_node4":{
   [junit4]   2>           
"core":"testNodeWithMultipleReplicasLost_shard2_replica_n4",
   [junit4]   2>           "leader":"true",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "node_name":"127.0.0.1:10000_solr",
   [junit4]   2>           "state":"active",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0},
   [junit4]   2>         "core_node5":{
   [junit4]   2>           
"core":"testNodeWithMultipleReplicasLost_shard2_replica_n5",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "node_name":"127.0.0.1:10004_solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       "range":"0-7fffffff",
   [junit4]   2>       "state":"active"},
   [junit4]   2>     "shard1":{
   [junit4]   2>       "replicas":{
   [junit4]   2>         "core_node1":{
   [junit4]   2>           
"core":"testNodeWithMultipleReplicasLost_shard1_replica_n1",
   [junit4]   2>           "leader":"true",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "node_name":"127.0.0.1:10003_solr",
   [junit4]   2>           "state":"active",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0},
   [junit4]   2>         "core_node2":{
   [junit4]   2>           
"core":"testNodeWithMultipleReplicasLost_shard1_replica_n2",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "node_name":"127.0.0.1:10005_solr",
   [junit4]   2>           "state":"active",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0},
   [junit4]   2>         "core_node3":{
   [junit4]   2>           
"core":"testNodeWithMultipleReplicasLost_shard1_replica_n3",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":10240,
   [junit4]   2>           "node_name":"127.0.0.1:10004_solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>           "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>       "range":"80000000-ffffffff",
   [junit4]   2>       "state":"active"}}}
   [junit4]   2> 343102 INFO  
(TEST-TestComputePlanAction.testNodeWithMultipleReplicasLost-seed#[B196C5F89CDD2326])
 [    ] o.a.s.SolrTestCaseJ4 ###Ending testNodeWithMultipleReplicasLost
   [junit4]   2> 343103 INFO  
(TEST-TestComputePlanAction.testNodeWithMultipleReplicasLost-seed#[B196C5F89CDD2326])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase #######################################
   [junit4]   2> ############ CLUSTER STATE ############
   [junit4]   2> #######################################
   [junit4]   2> ## Live nodes:         3
   [junit4]   2> ## Empty nodes:        0
   [junit4]   2> ## Dead nodes:         1
   [junit4]   2> ##             127.0.0.1:10004_solr
   [junit4]   2> ## Collections:        [.system, 
testNodeWithMultipleReplicasLost]
   [junit4]   2> ## Max replicas per node:      2
   [junit4]   2> ## Min replicas per node:      1
   [junit4]   2> ## Total replicas:             5
   [junit4]   2> ## * .system           1
   [junit4]   2> ##             - active           1
   [junit4]   2> ## * testNodeWithMultipleReplicasLost          4
   [junit4]   2> ##             - active           4
   [junit4]   2> ######### Solr op counts ##########
   [junit4]   2> ##             - CREATE             1
   [junit4]   2> ##             - autoscaling        3
   [junit4]   2> ##             - update             2
   [junit4]   2> ######### Autoscaling event counts ###########
   [junit4]   2> ## * Trigger: node_lost_trigger
   [junit4]   2> ##             - BEFORE_ACTION     1
   [junit4]   2> ##             - STARTED         1
   [junit4]   2> 
   [junit4]   2> 343109 INFO  
(TEST-TestComputePlanAction.testNodeAdded-seed#[B196C5F89CDD2326]) [    ] 
o.a.s.SolrTestCaseJ4 ###Starting testNodeAdded
   [junit4]   2> 343109 DEBUG 
(TEST-TestComputePlanAction.testNodeAdded-seed#[B196C5F89CDD2326]) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
11
   [junit4]   2> 343109 DEBUG 
(TEST-TestComputePlanAction.testNodeAdded-seed#[B196C5F89CDD2326]) [    ] 
o.a.s.c.a.s.SimNodeStateProvider - removing dead node values: 
127.0.0.1:10004_solr
   [junit4]   2> 343109 DEBUG 
(TEST-TestComputePlanAction.testNodeAdded-seed#[B196C5F89CDD2326]) [    ] 
o.a.s.c.a.s.SimNodeStateProvider --removing value for 127.0.0.1:10003_solr
   [junit4]   2> 343109 DEBUG 
(TEST-TestComputePlanAction.testNodeAdded-seed#[B196C5F89CDD2326]) [    ] 
o.a.s.c.a.s.SimNodeStateProvider --removing value for 127.0.0.1:10005_solr
   [junit4]   2> 343110 INFO  
(TEST-TestComputePlanAction.testNodeAdded-seed#[B196C5F89CDD2326]) [    ] 
o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing 
object cache...
   [junit4]   2> 343110 DEBUG 
(TEST-TestComputePlanAction.testNodeAdded-seed#[B196C5F89CDD2326]) [    ] 
o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 343110 DEBUG 
(TEST-TestComputePlanAction.testNodeAdded-seed#[B196C5F89CDD2326]) [    ] 
o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 343110 DEBUG 
(TEST-TestComputePlanAction.testNodeAdded-seed#[B196C5F89CDD2326]) [    ] 
o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 343110 WARN  (AutoscalingActionExecutor-3025-thread-1) [    ] 
o.a.s.c.a.SystemLogListener Exception sending event to collection .system
   [junit4]   2> java.io.IOException: java.lang.InterruptedException
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:632)
 ~[test/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.SystemLogListener.onEvent(SystemLogListener.java:118)
 [java/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerListeners.fireListeners(ScheduledTriggers.java:792)
 [java/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerListeners.fireListeners(ScheduledTriggers.java:768)
 [java/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$add$3(ScheduledTriggers.java:315)
 [java/:?]
   [junit4]   2>        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514) [?:?]
   [junit4]   2>        at 
java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
 [java/:?]
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135) 
[?:?]
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) 
[?:?]
   [junit4]   2>        at java.lang.Thread.run(Thread.java:844) [?:?]
   [junit4]   2> Caused by: java.lang.InterruptedException
   [junit4]   2>        at 
java.util.concurrent.FutureTask.awaitDone(FutureTask.java:418) ~[?:?]
   [junit4]   2>        at 
java.util.concurrent.FutureTask.get(FutureTask.java:190) ~[?:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:630)
 ~[test/:?]
   [junit4]   2>        ... 10 more
   [junit4]   2> 343112 WARN  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are 
closed, exiting.
   [junit4]   2> 343125 DEBUG (simCloudManagerPool-3023-thread-19) [    ] 
o.a.s.c.o.ClusterStateMutator building a new cName: .system
   [junit4]   2> 343159 DEBUG (simCloudManagerPool-3023-thread-20) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- submit leader election for .system / 
shard1
   [junit4]   2> 343159 DEBUG (simCloudManagerPool-3023-thread-21) [    ] 
o.a.s.c.a.s.SimClusterStateProvider Running leader election for .system / shard1
   [junit4]   2> 343159 DEBUG (simCloudManagerPool-3023-thread-21) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for .system / shard1: 
core_node3
   [junit4]   2> 343161 WARN  (AutoscalingActionExecutor-3025-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Unhandled exception executing actions
   [junit4]   2> org.apache.lucene.store.AlreadyClosedException: 
ScheduledTrigger node_lost_trigger has been closed.
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerWrapper.dequeue(ScheduledTriggers.java:548)
 ~[java/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$add$3(ScheduledTriggers.java:318)
 [java/:?]
   [junit4]   2>        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514) [?:?]
   [junit4]   2>        at 
java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
 [java/:?]
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135) 
[?:?]
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) 
[?:?]
   [junit4]   2>        at java.lang.Thread.run(Thread.java:844) [?:?]
   [junit4]   2> 343161 DEBUG (AutoscalingActionExecutor-3025-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing took 3908 ms for event 
id=3317f7bdd15e0Tagfrec5zzlm65qw4au9xvp1fr
   [junit4]   2> 343161 DEBUG 
(TEST-TestComputePlanAction.testNodeAdded-seed#[B196C5F89CDD2326]) [    ] 
o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool 
executor
   [junit4]   2> 343161 DEBUG 
(TEST-TestComputePlanAction.testNodeAdded-seed#[B196C5F89CDD2326]) [    ] 
o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 343161 DEBUG 
(TEST-TestComputePlanAction.testNodeAdded-seed#[B196C5F89CDD2326]) [    ] 
o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 343176 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and 
.scheduled_maintenance triggers
   [junit4]   2> 343177 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 12
   [junit4]   2> 343177 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 12, lastZnodeVersion -1
   [junit4]   2> 343177 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 12
   [junit4]   2> 343192 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial 
livenodes: [127.0.0.1:10000_solr]
   [junit4]   2> 343196 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 343196 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 12, lastZnodeVersion 12
   [junit4]   2> 343197 DEBUG (ScheduledTrigger-3027-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 1
   [junit4]   2> 343220 DEBUG (ScheduledTrigger-3027-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 1
   [junit4]   2> 343244 DEBUG (ScheduledTrigger-3027-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 1
   [junit4]   2> 343264 DEBUG (ScheduledTrigger-3027-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 1
   [junit4]   2> 343280 DEBUG (simCloudManagerPool-3026-thread-1) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 343281 DEBUG (simCloudManagerPool-3026-thread-1) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
13
   [junit4]   2> 343282 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 13
   [junit4]   2> 343282 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 343282 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 13, lastZnodeVersion 13
   [junit4]   2> 343283 DEBUG (simCloudManagerPool-3026-thread-2) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 343283 DEBUG (simCloudManagerPool-3026-thread-2) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
14
   [junit4]   2> 343283 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 14
   [junit4]   2> 343283 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 343283 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 14, lastZnodeVersion 14
   [junit4]   2> 343285 DEBUG (ScheduledTrigger-3027-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 1
   [junit4]   2> 343305 DEBUG (ScheduledTrigger-3027-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 1
   [junit4]   2> 343325 DEBUG (ScheduledTrigger-3027-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 1
   [junit4]   2> 343345 DEBUG (ScheduledTrigger-3027-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 1
   [junit4]   2> 343365 DEBUG (ScheduledTrigger-3027-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 1
   [junit4]   2> 343385 DEBUG (ScheduledTrigger-3027-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 1
   [junit4]   2> 343387 DEBUG (simCloudManagerPool-3026-thread-3) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 343387 DEBUG (simCloudManagerPool-3026-thread-3) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
15
   [junit4]   2> 343387 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 15
   [junit4]   2> 343387 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger - Initial 
livenodes: [127.0.0.1:10000_solr]
   [junit4]   2> 343387 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated 
with properties: {event=nodeAdded, waitFor=1, actions=[{name=compute_plan, 
class=solr.ComputePlanAction}, {name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestComputePlanAction$AssertingTriggerAction}],
 enabled=true}
   [junit4]   2> 343388 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 15, lastZnodeVersion 15
   [junit4]   2> 343388 DEBUG (ScheduledTrigger-3027-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 343388 DEBUG (ScheduledTrigger-3027-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 1
   [junit4]   2> 343389 DEBUG (simCloudManagerPool-3026-thread-4) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 343389 DEBUG (simCloudManagerPool-3026-thread-4) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
16
   [junit4]   2> 343390 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 16
   [junit4]   2> 343390 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 16, lastZnodeVersion 16
   [junit4]   2> 343398 DEBUG (simCloudManagerPool-3026-thread-5) [    ] 
o.a.s.c.o.ClusterStateMutator building a new cName: testNodeAdded
   [junit4]   2> 343406 DEBUG (ScheduledTrigger-3027-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 1
   [junit4]   2> 343408 DEBUG (ScheduledTrigger-3027-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 343408 DEBUG (ScheduledTrigger-3027-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 1
   [junit4]   2> 343409 WARN  (simCloudManagerPool-3026-thread-5) [    ] 
o.a.s.c.a.c.CreateCollectionCmd Specified number

[...truncated too long message...]

ent(SystemLogListener.java:118) [java/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerListeners.fireListeners(ScheduledTriggers.java:792)
 [java/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerListeners.fireListeners(ScheduledTriggers.java:768)
 [java/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$add$3(ScheduledTriggers.java:315)
 [java/:?]
   [junit4]   2>        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514) [?:?]
   [junit4]   2>        at 
java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
 [java/:?]
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135) 
[?:?]
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) 
[?:?]
   [junit4]   2>        at java.lang.Thread.run(Thread.java:844) [?:?]
   [junit4]   2> Caused by: java.lang.InterruptedException
   [junit4]   2>        at 
java.util.concurrent.FutureTask.awaitDone(FutureTask.java:418) ~[?:?]
   [junit4]   2>        at 
java.util.concurrent.FutureTask.get(FutureTask.java:190) ~[?:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:630)
 ~[test/:?]
   [junit4]   2>        ... 10 more
   [junit4]   2> 343508 WARN  (AutoscalingActionExecutor-3028-thread-1) [    ] 
o.a.s.c.a.SystemLogListener Exception sending event to collection .system
   [junit4]   2> java.io.IOException: java.util.concurrent.ExecutionException: 
java.io.IOException: already closed
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:632)
 ~[test/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.SystemLogListener.onEvent(SystemLogListener.java:118)
 [java/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerListeners.fireListeners(ScheduledTriggers.java:792)
 [java/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerListeners.fireListeners(ScheduledTriggers.java:759)
 [java/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$add$3(ScheduledTriggers.java:325)
 [java/:?]
   [junit4]   2>        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514) [?:?]
   [junit4]   2>        at 
java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
 [java/:?]
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135) 
[?:?]
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) 
[?:?]
   [junit4]   2>        at java.lang.Thread.run(Thread.java:844) [?:?]
   [junit4]   2> Caused by: java.util.concurrent.ExecutionException: 
java.io.IOException: already closed
   [junit4]   2>        at 
java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:?]
   [junit4]   2>        at 
java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[?:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:630)
 ~[test/:?]
   [junit4]   2>        ... 10 more
   [junit4]   2> Caused by: java.io.IOException: already closed
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.sim.SimClusterStateProvider.ensureNotClosed(SimClusterStateProvider.java:1755)
 ~[test/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.sim.SimClusterStateProvider.simUpdate(SimClusterStateProvider.java:1249)
 ~[test/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.simHandleSolrRequest(SimCloudManager.java:714)
 ~[test/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.lambda$request$15(SimCloudManager.java:629)
 ~[test/:?]
   [junit4]   2>        ... 5 more
   [junit4]   2> 343509 WARN  (AutoscalingActionExecutor-3028-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Unhandled exception executing actions
   [junit4]   2> org.apache.lucene.store.AlreadyClosedException: 
ScheduledTrigger node_added_trigger has been closed.
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerWrapper.dequeue(ScheduledTriggers.java:548)
 ~[java/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$add$3(ScheduledTriggers.java:318)
 [java/:?]
   [junit4]   2>        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514) [?:?]
   [junit4]   2>        at 
java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
 [java/:?]
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135) 
[?:?]
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) 
[?:?]
   [junit4]   2>        at java.lang.Thread.run(Thread.java:844) [?:?]
   [junit4]   2> 343509 DEBUG (AutoscalingActionExecutor-3028-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing took 914 ms for event 
id=33184e316a5e6Tagfrec5zzlm65qw4au9xvp1fw
   [junit4]   2> 343509 DEBUG 
(SUITE-TestComputePlanAction-seed#[B196C5F89CDD2326]-worker) [    ] 
o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool 
executor
   [junit4]   2> 343509 DEBUG 
(SUITE-TestComputePlanAction-seed#[B196C5F89CDD2326]-worker) [    ] 
o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 343509 DEBUG 
(SUITE-TestComputePlanAction-seed#[B196C5F89CDD2326]-worker) [    ] 
o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.sim.TestComputePlanAction_B196C5F89CDD2326-001
   [junit4]   2> Aug 26, 2018 5:05:21 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 3 leaked 
thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene80): {}, 
docValues:{}, maxPointsInLeafNode=502, maxMBSortInHeap=7.193789694687759, 
sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@38fc8249),
 locale=en-FM, timezone=Europe/Ljubljana
   [junit4]   2> NOTE: Linux 4.15.0-32-generic amd64/Oracle Corporation 10 
(64-bit)/cpus=8,threads=1,free=229434952,total=529530880
   [junit4]   2> NOTE: All tests run in this JVM: [BlockJoinFacetDistribTest, 
TestSmileRequest, CloudMLTQParserTest, TestJmxIntegration, 
PeerSyncWithIndexFingerprintCachingTest, TestGeoJSONResponseWriter, 
TestEmbeddedSolrServerSchemaAPI, TestClassicSimilarityFactory, 
TestHashPartitioner, TestManagedStopFilterFactory, NumericFieldsTest, 
DeleteStatusTest, TestLuceneIndexBackCompat, TestFieldResource, 
ReplicaListTransformerTest, TestSlowCompositeReaderWrapper, 
TestReloadAndDeleteDocs, WordBreakSolrSpellCheckerTest, TestPhraseSuggestions, 
TestSolrConfigHandlerConcurrent, CdcrRequestHandlerTest, 
TestGenericDistributedQueue, AddSchemaFieldsUpdateProcessorFactoryTest, 
DistributedDebugComponentTest, CachingDirectoryFactoryTest, 
DocExpirationUpdateProcessorFactoryTest, TestBackupRepositoryFactory, 
TestFilteredDocIdSet, CollectionStateFormat2Test, 
TestDynamicFieldCollectionResource, ManagedSchemaRoundRobinCloudTest, 
DataDrivenBlockJoinTest, RemoteQueryErrorTest, BufferStoreTest, 
TestCloudSchemaless, HLLSerializationTest, TestMultiWordSynonyms, 
TestMultiValuedNumericRangeQuery, TestComplexPhraseLeadingWildcard, 
ZkShardTermsTest, TestIntervalFaceting, CursorMarkTest, MetricsConfigTest, 
SpellPossibilityIteratorTest, TestFreeTextSuggestions, 
TestTolerantUpdateProcessorRandomCloud, TestNumericTokenStream, 
TestSolrQueryParser, TestRecoveryHdfs, AlternateDirectoryTest, TestConfig, 
LukeRequestHandlerTest, DistributedFacetPivotLargeTest, 
SpatialHeatmapFacetsTest, DisMaxRequestHandlerTest, TestLegacyFieldReuse, 
TestSweetSpotSimilarityFactory, TestSearchPerf, TestHalfAndHalfDocValues, 
TermVectorComponentTest, V2StandaloneTest, TestZkChroot, 
IndexBasedSpellCheckerTest, DirectoryFactoryTest, LegacyCloudClusterPropTest, 
TestAnalyzedSuggestions, SolrGraphiteReporterTest, TestCloudDeleteByQuery, 
CheckHdfsIndexTest, TestHighlightDedupGrouping, TestReRankQParserPlugin, 
AutoAddReplicasIntegrationTest, TestCustomSort, RequestLoggingTest, 
EmbeddedSolrNoSerializeTest, TestComputePlanAction]
   [junit4] Completed [174/830 (1!)] on J1 in 1.79s, 3 tests, 1 failure <<< 
FAILURES!

[...truncated 44541 lines...]
[repro] Jenkins log URL: 
https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/22748/consoleText

[repro] Revision: f26dd13b34e3d3a6921230cfe44ff34b2c319e7b

[repro] Ant options: "-Dargs=-XX:-UseCompressedOops -XX:+UseParallelGC"
[repro] Repro line:  ant test  -Dtestcase=TestComputePlanAction 
-Dtests.method=testNodeAdded -Dtests.seed=B196C5F89CDD2326 -Dtests.multiplier=3 
-Dtests.slow=true -Dtests.locale=en-FM -Dtests.timezone=Europe/Ljubljana 
-Dtests.asserts=true -Dtests.file.encoding=US-ASCII

[repro] ant clean

[...truncated 6 lines...]
[repro] Test suites by module:
[repro]    solr/core
[repro]       TestComputePlanAction
[repro] ant compile-test

[...truncated 3397 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=5 
-Dtests.class="*.TestComputePlanAction" -Dtests.showOutput=onerror 
"-Dargs=-XX:-UseCompressedOops -XX:+UseParallelGC" 
-Dtests.seed=B196C5F89CDD2326 -Dtests.multiplier=3 -Dtests.slow=true 
-Dtests.locale=en-FM -Dtests.timezone=Europe/Ljubljana -Dtests.asserts=true 
-Dtests.file.encoding=US-ASCII

[...truncated 86 lines...]
[repro] Failures:
[repro]   0/5 failed: 
org.apache.solr.cloud.autoscaling.sim.TestComputePlanAction
[repro] Exiting with code 0

[...truncated 40 lines...]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to