Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4981/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

3 tests failed.
FAILED:  
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testNodeLostTriggerRestoreState

Error Message:
The trigger did not fire at all

Stack Trace:
java.lang.AssertionError: The trigger did not fire at all
        at 
__randomizedtesting.SeedInfo.seed([E16C05018E6AFDBA:CA93D05A1412E86A]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.assertTrue(Assert.java:41)
        at 
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testNodeLostTriggerRestoreState(TestSimTriggerIntegration.java:332)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testNodeMarkersRegistration

Error Message:
Path /autoscaling/nodeAdded/127.0.0.1:10031_solr should have been deleted

Stack Trace:
java.lang.AssertionError: Path /autoscaling/nodeAdded/127.0.0.1:10031_solr 
should have been deleted
        at 
__randomizedtesting.SeedInfo.seed([E16C05018E6AFDBA:F9D68D0D805F3055]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.assertTrue(Assert.java:41)
        at org.junit.Assert.assertFalse(Assert.java:64)
        at 
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testNodeMarkersRegistration(TestSimTriggerIntegration.java:891)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testNodeMarkersRegistration

Error Message:
Path /autoscaling/nodeAdded/127.0.0.1:10059_solr should have been deleted

Stack Trace:
java.lang.AssertionError: Path /autoscaling/nodeAdded/127.0.0.1:10059_solr 
should have been deleted
        at 
__randomizedtesting.SeedInfo.seed([E16C05018E6AFDBA:F9D68D0D805F3055]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.assertTrue(Assert.java:41)
        at org.junit.Assert.assertFalse(Assert.java:64)
        at 
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testNodeMarkersRegistration(TestSimTriggerIntegration.java:891)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 12528 lines...]
   [junit4] Suite: 
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration
   [junit4]   2> 8685 INFO  
(SUITE-TestSimTriggerIntegration-seed#[E16C05018E6AFDBA]-worker) [    ] 
o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: 
test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: 
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.sim.TestSimTriggerIntegration_E16C05018E6AFDBA-001/init-core-data-001
   [junit4]   2> 9227 INFO  
(SUITE-TestSimTriggerIntegration-seed#[E16C05018E6AFDBA]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) 
w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 9436 INFO  
(SUITE-TestSimTriggerIntegration-seed#[E16C05018E6AFDBA]-worker) [    ] 
o.e.j.u.log Logging initialized @9522ms to org.eclipse.jetty.util.log.Slf4jLog
   [junit4]   2> 9468 INFO  
(SUITE-TestSimTriggerIntegration-seed#[E16C05018E6AFDBA]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: 
@org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN) 
w/ MAC_OS_X supressed clientAuth
   [junit4] IGNOR/A 0.03s J0 | 
TestSimTriggerIntegration.testNodeMarkersRegistration
   [junit4]    > Assumption #1: 'badapple' test group is disabled 
(@BadApple(bugUrl=https://issues.apache.org/jira/browse/SOLR-12028))
   [junit4] IGNOR/A 0.00s J0 | 
TestSimTriggerIntegration.testEventFromRestoredState
   [junit4]    > Assumption #1: 'badapple' test group is disabled 
(@BadApple(bugUrl=https://issues.apache.org/jira/browse/SOLR-12028))
   [junit4] IGNOR/A 0.00s J0 | TestSimTriggerIntegration.testNodeAddedTrigger
   [junit4]    > Assumption #1: 'badapple' test group is disabled 
(@BadApple(bugUrl=https://issues.apache.org/jira/browse/SOLR-12028))
   [junit4]   2> 9724 INFO  
(TEST-TestSimTriggerIntegration.testListeners-seed#[E16C05018E6AFDBA]) [    ] 
o.a.s.SolrTestCaseJ4 ###Starting testListeners
   [junit4]   2> 10124 INFO  
(TEST-TestSimTriggerIntegration.testListeners-seed#[E16C05018E6AFDBA]) [    ] 
o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find 
system property or JNDI)
   [junit4]   2> 10876 DEBUG 
(TEST-TestSimTriggerIntegration.testListeners-seed#[E16C05018E6AFDBA]) [    ] 
o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader: 
127.0.0.1:10000_solr
   [junit4]   2> 10932 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and 
.scheduled_maintenance triggers
   [junit4]   2> 10940 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 1
   [junit4]   2> 11026 DEBUG 
(TEST-TestSimTriggerIntegration.testListeners-seed#[E16C05018E6AFDBA]) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, 
currentVersion=0
   [junit4]   2> 11034 DEBUG 
(TEST-TestSimTriggerIntegration.testListeners-seed#[E16C05018E6AFDBA]) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 0
   [junit4]   2> 11034 INFO  
(TEST-TestSimTriggerIntegration.testListeners-seed#[E16C05018E6AFDBA]) [    ] 
o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history 
in memory.
   [junit4]   2> 11539 DEBUG (simCloudManagerPool-2-thread-2) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 11648 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 0
   [junit4]   2> 11648 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 11648 DEBUG (simCloudManagerPool-2-thread-2) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
2
   [junit4]   2> 11673 DEBUG (simCloudManagerPool-2-thread-3) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 11673 DEBUG (simCloudManagerPool-2-thread-3) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
3
   [junit4]   2> 11679 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial 
livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
   [junit4]   2> 11683 DEBUG (ScheduledTrigger-3-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 11685 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 11685 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 1
   [junit4]   2> 11685 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 11686 INFO  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.s.TestSimTriggerIntegration TestTriggerAction init
   [junit4]   2> 11687 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger - Initial 
livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
   [junit4]   2> 11687 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated 
with properties: {event=nodeAdded, waitFor=3, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration$TestTriggerAction},
 {name=test1, 
class=org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration$TestDummyAction}],
 enabled=true}
   [junit4]   2> 11688 DEBUG (ScheduledTrigger-3-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 11688 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 11688 DEBUG (ScheduledTrigger-3-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 11688 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 11691 DEBUG (simCloudManagerPool-2-thread-4) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 11692 DEBUG (simCloudManagerPool-2-thread-4) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
4
   [junit4]   2> 11693 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 11694 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 11694 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
   [junit4]   2> 11695 DEBUG (simCloudManagerPool-2-thread-5) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 11696 DEBUG (simCloudManagerPool-2-thread-5) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
5
   [junit4]   2> 11698 DEBUG 
(TEST-TestSimTriggerIntegration.testListeners-seed#[E16C05018E6AFDBA]) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- creating marker: 
/autoscaling/nodeAdded/127.0.0.1:10002_solr
   [junit4]   2> 11698 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
   [junit4]   2> 11698 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 11698 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
   [junit4]   2> 11706 DEBUG (ScheduledTrigger-3-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3 and last live nodes: 2
   [junit4]   2> 11710 DEBUG (ScheduledTrigger-3-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 11710 DEBUG (ScheduledTrigger-3-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 11711 DEBUG (ScheduledTrigger-3-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10002_solr at time 
397709702241679
   [junit4]   2> 11731 DEBUG (ScheduledTrigger-3-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 11732 DEBUG (ScheduledTrigger-3-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 11732 DEBUG (ScheduledTrigger-3-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 11753 DEBUG (ScheduledTrigger-3-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 11753 DEBUG (ScheduledTrigger-3-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 11753 DEBUG (ScheduledTrigger-3-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 11784 DEBUG (ScheduledTrigger-3-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 11784 DEBUG (ScheduledTrigger-3-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 11784 DEBUG (ScheduledTrigger-3-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 11784 DEBUG (ScheduledTrigger-3-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing 
registered processor for nodes: [127.0.0.1:10002_solr] added at times 
[397709702241679], now=397713370883429
   [junit4]   2> 11786 DEBUG (ScheduledTrigger-3-thread-2) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"169b701869d8fT33tdptpspk9vvtgfep2dubbki",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":397709702241679,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[397709702241679],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "nodeNames":["127.0.0.1:10002_solr"]}}
   [junit4]   2> 11786 DEBUG (ScheduledTrigger-3-thread-2) [    ] 
o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, 
node_added_trigger]
   [junit4]   2> 11794 DEBUG (simCloudManagerPool-2-thread-6) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- simCreateCollection .system, 
currentVersion=1
   [junit4]   2> 11814 DEBUG (simCloudManagerPool-2-thread-6) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, 
currentVersion=1
   [junit4]   2> 11815 DEBUG (simCloudManagerPool-2-thread-6) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 1
   [junit4]   2> 11874 DEBUG (simCloudManagerPool-2-thread-6) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, 
currentVersion=2
   [junit4]   2> 11879 DEBUG (simCloudManagerPool-2-thread-6) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 2
   [junit4]   2> 12005 DEBUG (simCloudManagerPool-2-thread-7) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, 
currentVersion=3
   [junit4]   2> 12006 DEBUG (simCloudManagerPool-2-thread-7) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 3
   [junit4]   2> 12025 DEBUG (simCloudManagerPool-2-thread-8) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, 
currentVersion=4
   [junit4]   2> 12026 DEBUG (simCloudManagerPool-2-thread-8) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 4
   [junit4]   2> 12037 DEBUG (simCloudManagerPool-2-thread-9) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, 
currentVersion=5
   [junit4]   2> 12038 DEBUG (simCloudManagerPool-2-thread-9) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 5
   [junit4]   2> 12048 DEBUG (simCloudManagerPool-2-thread-12) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for .system / shard1 
(currentVersion=6): {"core_node1":{
   [junit4]   2>     "core":".system_shard1_replica_n1",
   [junit4]   2>     "shard":"shard1",
   [junit4]   2>     "collection":".system",
   [junit4]   2>     "node_name":"127.0.0.1:10002_solr",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":10240,
   [junit4]   2>     "state":"active",
   [junit4]   2>     "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>     "SEARCHER.searcher.numDocs":0}}
   [junit4]   2> 12055 DEBUG (simCloudManagerPool-2-thread-6) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- finished createCollection .system, 
currentVersion=6
   [junit4]   2> 12060 DEBUG (simCloudManagerPool-2-thread-6) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, 
currentVersion=6
   [junit4]   2> 12060 DEBUG (simCloudManagerPool-2-thread-6) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 6
   [junit4]   2> 12070 DEBUG (ScheduledTrigger-3-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Removing new node from tracking: 127.0.0.1:10002_solr
   [junit4]   2> 12070 DEBUG (AutoscalingActionExecutor-4-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"169b701869d8fT33tdptpspk9vvtgfep2dubbki",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":397709702241679,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[397709702241679],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "_enqueue_time_":397713516866029,
   [junit4]   2>     "nodeNames":["127.0.0.1:10002_solr"]}}
   [junit4]   2> 12082 DEBUG (AutoscalingActionExecutor-4-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
   [junit4]   2> 12082 DEBUG (AutoscalingActionExecutor-4-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger after 100ms
   [junit4]   2> 12082 DEBUG (AutoscalingActionExecutor-4-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing took 619 ms for event 
id=169b701869d8fT33tdptpspk9vvtgfep2dubbki
   [junit4]   2> 12141 DEBUG 
(TEST-TestSimTriggerIntegration.testListeners-seed#[E16C05018E6AFDBA]) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- creating marker: 
/autoscaling/nodeAdded/127.0.0.1:10003_solr
   [junit4]   2> 12184 DEBUG (ScheduledTrigger-3-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 12184 DEBUG (ScheduledTrigger-3-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4 and last live nodes: 3
   [junit4]   2> 12184 DEBUG (ScheduledTrigger-3-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 12184 DEBUG (ScheduledTrigger-3-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10003_solr at time 
397733380341229
   [junit4]   2> 12214 DEBUG (ScheduledTrigger-3-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 12214 DEBUG (ScheduledTrigger-3-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4 and last live nodes: 4
   [junit4]   2> 12214 DEBUG (ScheduledTrigger-3-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 12238 DEBUG (ScheduledTrigger-3-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4 and last live nodes: 4
   [junit4]   2> 12239 DEBUG (ScheduledTrigger-3-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 12239 DEBUG (ScheduledTrigger-3-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 12268 DEBUG (ScheduledTrigger-3-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4 and last live nodes: 4
   [junit4]   2> 12268 DEBUG (ScheduledTrigger-3-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 12268 DEBUG (ScheduledTrigger-3-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 12268 DEBUG (ScheduledTrigger-3-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing 
registered processor for nodes: [127.0.0.1:10003_solr] added at times 
[397733380341229], now=397737574963329
   [junit4]   2> 12268 DEBUG (ScheduledTrigger-3-thread-2) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"169bc84d9bdedT33tdptpspk9vvtgfep2dubbkl",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":397733380341229,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[397733380341229],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "nodeNames":["127.0.0.1:10003_solr"]}}
   [junit4]   2> 12268 DEBUG (ScheduledTrigger-3-thread-2) [    ] 
o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, 
node_added_trigger]
   [junit4]   2> 12273 DEBUG (ScheduledTrigger-3-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Removing new node from tracking: 127.0.0.1:10003_solr
   [junit4]   2> 12274 DEBUG (AutoscalingActionExecutor-4-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"169bc84d9bdedT33tdptpspk9vvtgfep2dubbkl",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":397733380341229,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[397733380341229],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "_enqueue_time_":397737599438429,
   [junit4]   2>     "nodeNames":["127.0.0.1:10003_solr"]}}
   [junit4]   2> 12283 WARN  (AutoscalingActionExecutor-4-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Exception executing actions
   [junit4]   2> org.apache.solr.cloud.autoscaling.TriggerActionException: 
Error processing action for trigger event: {
   [junit4]   2>   "id":"169bc84d9bdedT33tdptpspk9vvtgfep2dubbkl",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":397733380341229,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[397733380341229],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "_enqueue_time_":397737599438429,
   [junit4]   2>     "nodeNames":["127.0.0.1:10003_solr"]}}
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:314)
 ~[java/:?]
   [junit4]   2>        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_172]
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
 ~[java/:?]
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
[?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
[?:1.8.0_172]
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
   [junit4]   2> Caused by: java.lang.RuntimeException: failure
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration$TestDummyAction.process(TestSimTriggerIntegration.java:942)
 ~[test/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:311)
 ~[java/:?]
   [junit4]   2>        ... 6 more
   [junit4]   2> 12300 DEBUG (AutoscalingActionExecutor-4-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
   [junit4]   2> 12300 DEBUG (AutoscalingActionExecutor-4-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger after 100ms
   [junit4]   2> 12300 DEBUG (AutoscalingActionExecutor-4-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing took 1348 ms for event 
id=169bc84d9bdedT33tdptpspk9vvtgfep2dubbkl
   [junit4]   2> 12401 DEBUG (ScheduledTrigger-3-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4 and last live nodes: 4
   [junit4]   2> 12401 DEBUG (ScheduledTrigger-3-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 12401 DEBUG (ScheduledTrigger-3-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 12409 DEBUG 
(TEST-TestSimTriggerIntegration.testListeners-seed#[E16C05018E6AFDBA]) [    ] 
o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 12409 INFO  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are 
closed, exiting.
   [junit4]   2> 12409 DEBUG 
(TEST-TestSimTriggerIntegration.testListeners-seed#[E16C05018E6AFDBA]) [    ] 
o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 12410 DEBUG 
(TEST-TestSimTriggerIntegration.testListeners-seed#[E16C05018E6AFDBA]) [    ] 
o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 12411 DEBUG 
(TEST-TestSimTriggerIntegration.testListeners-seed#[E16C05018E6AFDBA]) [    ] 
o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool 
executor
   [junit4]   2> 12411 DEBUG 
(TEST-TestSimTriggerIntegration.testListeners-seed#[E16C05018E6AFDBA]) [    ] 
o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 12411 DEBUG 
(TEST-TestSimTriggerIntegration.testListeners-seed#[E16C05018E6AFDBA]) [    ] 
o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 12412 INFO  
(TEST-TestSimTriggerIntegration.testListeners-seed#[E16C05018E6AFDBA]) [    ] 
o.a.s.SolrTestCaseJ4 ###Ending testListeners
   [junit4]   2> 12471 INFO  
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[E16C05018E6AFDBA])
 [    ] o.a.s.SolrTestCaseJ4 ###Starting testNodeLostTriggerRestoreState
   [junit4]   2> 12494 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[E16C05018E6AFDBA])
 [    ] o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader: 
127.0.0.1:10004_solr
   [junit4]   2> 12494 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and 
.scheduled_maintenance triggers
   [junit4]   2> 12494 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[E16C05018E6AFDBA])
 [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, 
currentVersion=0
   [junit4]   2> 12494 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[E16C05018E6AFDBA])
 [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 0
   [junit4]   2> 12495 INFO  
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[E16C05018E6AFDBA])
 [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics 
history in memory.
   [junit4]   2> 12495 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 1
   [junit4]   2> 12496 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 0
   [junit4]   2> 12496 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 12497 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial 
livenodes: [127.0.0.1:10004_solr, 127.0.0.1:10005_solr]
   [junit4]   2> 12497 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 12497 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 12498 DEBUG (ScheduledTrigger-9-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12500 DEBUG (simCloudManagerPool-8-thread-2) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 12500 DEBUG (simCloudManagerPool-8-thread-2) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
2
   [junit4]   2> 12501 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 12501 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 12502 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
   [junit4]   2> 12503 DEBUG (simCloudManagerPool-8-thread-3) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 12504 DEBUG (simCloudManagerPool-8-thread-3) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
3
   [junit4]   2> 12505 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 12506 INFO  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.s.TestSimTriggerIntegration TestTriggerAction init
   [junit4]   2> 12506 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeLostTrigger NodeLostTrigger node_lost_restore_trigger - Initial 
livenodes: [127.0.0.1:10004_solr, 127.0.0.1:10005_solr]
   [junit4]   2> 12506 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[E16C05018E6AFDBA])
 [    ] o.a.s.c.a.s.SimClusterStateProvider  -- created marker: 
/autoscaling/nodeLost/127.0.0.1:10006_solr
   [junit4]   2> 12506 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 12506 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 12506 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12522 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12528 DEBUG (ScheduledTrigger-9-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12544 DEBUG (ScheduledTrigger-9-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12550 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12550 DEBUG (simCloudManagerPool-8-thread-4) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 12551 DEBUG (simCloudManagerPool-8-thread-4) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
4
   [junit4]   2> 12552 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 12552 INFO  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.s.TestSimTriggerIntegration TestTriggerAction init
   [junit4]   2> 12552 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeLostTrigger NodeLostTrigger node_lost_restore_trigger - Initial 
livenodes: [127.0.0.1:10004_solr, 127.0.0.1:10005_solr]
   [junit4]   2> 12553 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 12553 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
   [junit4]   2> 12553 DEBUG (ScheduledTrigger-9-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12568 DEBUG (ScheduledTrigger-9-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12573 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12589 DEBUG (ScheduledTrigger-9-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12595 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12616 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12616 DEBUG (ScheduledTrigger-9-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12644 DEBUG (ScheduledTrigger-9-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12644 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12672 DEBUG (ScheduledTrigger-9-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12672 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12692 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12693 DEBUG (ScheduledTrigger-9-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12717 DEBUG (ScheduledTrigger-9-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12717 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12747 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12747 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12777 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12777 DEBUG (ScheduledTrigger-9-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12805 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12805 DEBUG (ScheduledTrigger-9-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12826 DEBUG (ScheduledTrigger-9-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12826 DEBUG (ScheduledTrigger-9-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12847 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12847 DEBUG (ScheduledTrigger-9-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12867 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12867 DEBUG (ScheduledTrigger-9-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12895 DEBUG (ScheduledTrigger-9-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12895 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12918 DEBUG (ScheduledTrigger-9-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12918 DEBUG (ScheduledTrigger-9-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12939 DEBUG (ScheduledTrigger-9-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12939 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12969 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12969 DEBUG (ScheduledTrigger-9-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12989 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 12989 DEBUG (ScheduledTrigger-9-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13017 DEBUG (ScheduledTrigger-9-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13017 DEBUG (ScheduledTrigger-9-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13038 DEBUG (ScheduledTrigger-9-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13038 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13064 DEBUG (ScheduledTrigger-9-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13064 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13089 DEBUG (ScheduledTrigger-9-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13089 DEBUG (ScheduledTrigger-9-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13112 DEBUG (ScheduledTrigger-9-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13112 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13137 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13137 DEBUG (ScheduledTrigger-9-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13166 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13166 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13191 DEBUG (ScheduledTrigger-9-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13191 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13219 DEBUG (ScheduledTrigger-9-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13219 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13245 DEBUG (ScheduledTrigger-9-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13245 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13265 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13265 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13291 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13291 DEBUG (ScheduledTrigger-9-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13313 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13313 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13334 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13334 DEBUG (ScheduledTrigger-9-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13354 DEBUG (ScheduledTrigger-9-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13354 DEBUG (ScheduledTrigger-9-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13376 DEBUG (ScheduledTrigger-9-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13376 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13397 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13397 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13426 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13426 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13447 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13447 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13471 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13471 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13494 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13494 DEBUG (ScheduledTrigger-9-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13525 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13525 DEBUG (ScheduledTrigger-9-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13554 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13554 DEBUG (ScheduledTrigger-9-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13581 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13581 DEBUG (ScheduledTrigger-9-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13610 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13610 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13630 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13630 DEBUG (ScheduledTrigger-9-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13654 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13655 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13682 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13683 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13704 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13704 DEBUG (ScheduledTrigger-9-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13735 DEBUG (ScheduledTrigger-9-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13735 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13758 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13758 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13778 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13779 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13800 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13800 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13822 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13822 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13844 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13844 DEBUG (ScheduledTrigger-9-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13874 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13874 DEBUG (ScheduledTrigger-9-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13905 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13905 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13925 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13925 DEBUG (ScheduledTrigger-9-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13956 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13956 DEBUG (ScheduledTrigger-9-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13986 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 13986 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 14016 DEBUG (ScheduledTrigger-9-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 14016 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 14046 DEBUG (ScheduledTrigger-9-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 14046 DEBUG (ScheduledTrigger-9-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 14075 DEBUG (ScheduledTrigger-9-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 14075 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 14103 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 14103 DEBUG (ScheduledTrigger-9-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 14132 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 14132 DEBUG (ScheduledTrigger-9-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 14160 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 14160 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 14184 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 14184 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 14214 DEBUG (ScheduledTrigger-9-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 14214 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 14237 DEBUG (ScheduledTrigger-9-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 14237 DEBUG (ScheduledTrigger-9-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 14267 DEBUG (ScheduledTrigger-9-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 14267 DEBUG (ScheduledTrigger-9-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 14297 DEBUG (ScheduledTrigger-9-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 14297 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 14324 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 14324 DEBUG (ScheduledTrigger-9-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 14352 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 14352 DEBUG (ScheduledTrigger-9-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 14356 INFO  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are 
closed, exiting.
   [junit4]   2> 14356 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[E16C05018E6AFDBA])
 [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool 
executor now
   [junit4]   2> 14356 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[E16C05018E6AFDBA])
 [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 14356 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[E16C05018E6AFDBA])
 [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 14357 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[E16C05018E6AFDBA])
 [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread 
pool executor
   [junit4]   2> 14358 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[E16C05018E6AFDBA])
 [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 14358 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[E16C05018E6AFDBA])
 [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed 
explicitly
   [junit4]   2> 14359 INFO  
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[E16C05018E6AFDBA])
 [    ] o.a.s.SolrTestCaseJ4 ###Ending testNodeLostTriggerRestoreState
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=TestSimTriggerIntegration 
-Dtests.method=testNodeLostTriggerRestoreState -Dtests.seed=E16C05018E6AFDBA 
-Dtests.slow=true -Dtests.locale=es-BO -Dtests.timezone=Asia/Kuala_Lumpur 
-Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE 1.91s J0 | 
TestSimTriggerIntegration.testNodeLostTriggerRestoreState <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: The trigger did not 
fire at all
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([E16C05018E6AFDBA:CA93D05A1412E86A]:0)
   [junit4]    >        at 
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testNodeLostTriggerRestoreState(TestSimTriggerIntegration.java:332)
   [junit4]    >        at java.lang.Thread.run(Thread.java:748)
   [junit4] IGNOR/A 0.00s J0 | TestSimTriggerIntegration.testCooldown
   [junit4]    > Assumption #1: 'badapple' test group is disabled 
(@BadApple(bugUrl=https://issues.apache.org/jira/browse/SOLR-12028))
   [junit4]   2> 14380 INFO  
(TEST-TestSimTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[E16C05018E6AFDBA])
 [    ] o.a.s.SolrTestCaseJ4 ###Starting testNodeAddedTriggerRestoreState
   [junit4]   2> 14386 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[E16C05018E6AFDBA])
 [    ] o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader: 
127.0.0.1:10007_solr
   [junit4]   2> 14386 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and 
.scheduled_maintenance triggers
   [junit4]   2> 14387 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[E16C05018E6AFDBA])
 [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, 
currentVersion=0
   [junit4]   2> 14387 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 1
   [junit4]   2> 14387 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[E16C05018E6AFDBA])
 [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 0
   [junit4]   2> 14387 INFO  
(TEST-TestSimTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[E16C05018E6AFDBA])
 [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics 
history in memory.
   [junit4]   2> 14388 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 0
   [junit4]   2> 14388 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 14389 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial 
livenodes: [127.0.0.1:10008_solr, 127.0.0.1:10007_solr]
   [junit4]   2> 14390 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 14390 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 14390 DEBUG (ScheduledTrigger-15-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 14393 DEBUG (simCloudManagerPool-14-thread-2) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 14393 DEBUG (simCloudManagerPool-14-thread-2) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
2
   [junit4]   2> 14395 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 14397 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 14398 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
   [junit4]   2> 14399 DEBUG (simCloudManagerPool-14-thread-3) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 14400 DEBUG (simCloudManagerPool-14-thread-3) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
3
   [junit4]   2> 14402 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 14403 INFO  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.s.TestSimTriggerIntegration TestTriggerAction init
   [junit4]   2> 14403 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_restore_trigger - 
Initial livenodes: [127.0.0.1:10008_solr, 127.0.0.1:10007_solr]
   [junit4]   2> 14403 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_restore_trigger 
instantiated with properties: {event=nodeAdded, waitFor=5, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration$TestTriggerAction}],
 enabled=true}
   [junit4]   2> 14403 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 14403 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 14404 DEBUG (ScheduledTrigger-15-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
   [junit4]   2> 14404 DEBUG (ScheduledTrigger-15-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 14406 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[E16C05018E6AFDBA])
 [    ] o.a.s.c.a.s.SimClusterStateProvider -- creating marker: 
/autoscaling/nodeAdded/127.0.0.1:10009_solr
   [junit4]   2> 14414 DEBUG (ScheduledTrigger-15-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3 and last live nodes: 2
   [junit4]   2> 14426 DEBUG (ScheduledTrigger-15-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
   [junit4]   2> 14426 DEBUG (ScheduledTrigger-15-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 14426 DEBUG (ScheduledTrigger-15-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10009_solr at time 
397845485168279
   [junit4]   2> 14439 DEBUG (ScheduledTrigger-15-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 14442 DEBUG (simCloudManagerPool-14-thread-4) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 14442 DEBUG (simCloudManagerPool-14-thread-4) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
4
   [junit4]   2> 14443 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 14443 INFO  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.s.TestSimTriggerIntegration TestTriggerAction init
   [junit4]   2> 14443 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_restore_trigger - 
Initial livenodes: [127.0.0.1:10009_solr, 127.0.0.1:10008_solr, 
127.0.0.1:10007_solr]
   [junit4]   2> 14443 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_restore_trigger 
instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration$TestTriggerAction}],
 enabled=true}
   [junit4]   2> 14444 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 14444 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
   [junit4]   2> 14444 DEBUG (ScheduledTrigger-15-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
   [junit4]   2> 14444 DEBUG (ScheduledTrigger-15-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 14444 DEBUG (ScheduledTrigger-15-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_restore_trigger firing 
registered processor for nodes: [127.0.0.1:10009_solr] added at times 
[397845485168279], now=397846392520879
   [junit4]   2> 14445 DEBUG (ScheduledTrigger-15-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"169d69ed1a697T33tdptpspk9vvtgfep2dubbkq",
   [junit4]   2>   "source":"node_added_restore_trigger",
   [junit4]   2>   "eventTime":397845485168279,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[397845485168279],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "nodeNames":["127.0.0.1:10009_solr"]}}
   [junit4]   2> 14445 DEBUG (ScheduledTrigger-15-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, 
node_added_restore_trigger]
   [junit4]   2> 14447 DEBUG (simCloudManagerPool-14-thread-5) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- simCreateCollection .system, 
currentVersion=1
   [junit4]   2> 14447 DEBUG (simCloudManagerPool-14-thread-5) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, 
currentVersion=1
   [junit4]   2> 14447 DEBUG (simCloudManagerPool-14-thread-5) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 1
   [junit4]   2> 14461 DEBUG (simCloudManagerPool-14-thread-5) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, 
currentVersion=2
   [junit4]   2> 14462 DEBUG (simCloudManagerPool-14-thread-5) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 2
   [junit4]   2> 14476 DEBUG (simCloudManagerPool-14-thread-6) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, 
currentVersion=3
   [junit4]   2> 14478 DEBUG (simCloudManagerPool-14-thread-6) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 3
   [junit4]   2> 14482 DEBUG (simCloudManagerPool-14-thread-9) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for .system / shard1 
(currentVersion=4): {"core_node1":{
   [junit4]   2>     "core":".system_shard1_replica_n1",
   [junit4]   2>     "shard":"shard1",
   [junit4]   2>     "collection":".system",
   [junit4]   2>     "node_name":"127.0.0.1:10007_solr",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":10240,
   [junit4]   2>     "state":"active",
   [junit4]   2>     "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>     "SEARCHER.searcher.numDocs":0}}
   [junit4]   2> 14496 DEBUG (simCloudManagerPool-14-thread-7) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, 
currentVersion=4
   [junit4]   2> 14496 DEBUG (simCloudManagerPool-14-thread-7) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 4
   [junit4]   2> 14512 DEBUG (simCloudManagerPool-14-thread-8) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, 
currentVersion=5
   [junit4]   2> 14512 DEBUG (simCloudManagerPool-14-thread-8) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 5
   [junit4]   2> 14513 DEBUG (simCloudManagerPool-14-thread-5) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- finished createCollection .system, 
currentVersion=6
   [junit4]   2> 14514 DEBUG (ScheduledTrigger-15-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Removing new node from tracking: 127.0.0.1:10009_solr
   [junit4]   2> 14514 DEBUG (AutoscalingActionExecutor-16-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"169d69ed1a697T33tdptpspk9vvtgfep2dubbkq",
   [junit4]   2>   "source":"node_added_restore_trigger",
   [junit4]   2>   "eventTime":397845485168279,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[397845485168279],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "_enqueue_time_":397846442423629,
   [junit4]   2>     "nodeNames":["127.0.0.1:10009_solr"]}}
   [junit4]   2> 14516 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[E16C05018E6AFDBA])
 [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool 
executor now
   [junit4]   2> 14516 INFO  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are 
closed, exiting.
   [junit4]   2> 14519 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[E16C05018E6AFDBA])
 [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 14519 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[E16C05018E6AFDBA])
 [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 14517 WARN  (AutoscalingActionExecutor-16-thread-1) [    ] 
o.a.s.c.a.SystemLogListener Exception sending event to collection .system
   [junit4]   2> java.io.IOException: java.util.concurrent.ExecutionException: 
java.io.IOException: already closed
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:654)
 ~[test/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.SystemLogListener.onEvent(SystemLogListener.java:118)
 ~[java/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerListeners.fireListeners(ScheduledTriggers.java:806)
 ~[java/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerListeners.fireListeners(ScheduledTriggers.java:773)
 ~[java/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:324)
 ~[java/:?]
   [junit4]   2>        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_172]
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
 ~[java/:?]
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
[?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
[?:1.8.0_172]
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
   [junit4]   2> Caused by: java.util.concurrent.ExecutionException: 
java.io.IOException: already closed
   [junit4]   2>        at 
java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_172]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:652)
 ~[test/:?]
   [junit4]   2>        ... 10 more
   [junit4]   2> Caused by: java.io.IOException: already closed
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.sim.SimClusterStateProvider.ensureNotClosed(SimClusterStateProvider.java:2208)
 ~[test/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.sim.SimClusterStateProvider.simUpdate(SimClusterStateProvider.java:1425)
 ~[test/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.simHandleSolrRequest(SimCloudManager.java:741)
 ~[test/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.lambda$request$15(SimCloudManager.java:651)
 ~[test/:?]
   [junit4]   2>        ... 5 more
   [junit4]   2> 14519 DEBUG (AutoscalingActionExecutor-16-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing took 262 ms for event 
id=169d69ed1a697T33tdptpspk9vvtgfep2dubbkq
   [junit4]   2> 14520 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[E16C05018E6AFDBA])
 [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread 
pool executor
   [junit4]   2> 14520 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[E16C05018E6AFDBA])
 [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 14520 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[E16C05018E6AFDBA])
 [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed 
explicitly
   [junit4]   2> 14523 INFO  
(TEST-TestSimTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[E16C05018E6AFDBA])
 [    ] o.a.s.SolrTestCaseJ4 ###Ending testNodeAddedTriggerRestoreState
   [junit4] IGNOR/A 0.00s J0 | TestSimTriggerIntegration.testNodeLostTrigger
   [junit4]    > Assumption #1: 'awaitsfix' test group is disabled 
(@AwaitsFix(bugUrl=https://issues.apache.org/jira/browse/SOLR-12028))
   [junit4] IGNOR/A 0.00s J0 | TestSimTriggerIntegration.testSearchRate
   [junit4]    > Assumption #1: 'awaitsfix' test group is disabled 
(@AwaitsFix(bugUrl=https://issues.apache.org/jira/browse/SOLR-12028))
   [junit4] IGNOR/A 0.00s J0 | TestSimTriggerIntegration.testEventQueue
   [junit4]    > Assumption #1: 'awaitsfix' test group is disabled 
(@AwaitsFix(bugUrl=https://issues.apache.org/jira/browse/SOLR-12028))
   [junit4] IGNOR/A 0.00s J0 | TestSimTriggerIntegration.testTriggerThrottling
   [junit4]    > Assumption #1: 'badapple' test group is disabled 
(@BadApple(bugUrl=https://issues.apache.org/

[...truncated too long message...]

utoscaling.ScheduledTriggers.lambda$add$4(ScheduledTriggers.java:284) ~[java/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.NodeLostTrigger.run(NodeLostTrigger.java:185) 
[java/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerWrapper.run(ScheduledTriggers.java:603)
 [java/:?]
   [junit4]   2>        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
[?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
 [?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
 [?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
[?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
[?:1.8.0_172]
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
   [junit4]   2> Caused by: java.lang.InterruptedException
   [junit4]   2>        at 
java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404) ~[?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[?:1.8.0_172]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:652)
 ~[test/:?]
   [junit4]   2>        ... 13 more
   [junit4]   2> 21878 DEBUG 
(TEST-TestSimTriggerIntegration.testTriggerThrottling-seed#[E16C05018E6AFDBA]) 
[    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread 
pool executor
   [junit4]   2> 21881 WARN  (ScheduledTrigger-162-thread-3) [    ] 
o.a.s.c.a.SystemLogListener Exception sending event to collection .system
   [junit4]   2> java.io.IOException: java.util.concurrent.ExecutionException: 
java.io.IOException: already closed
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:654)
 ~[test/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.SystemLogListener.onEvent(SystemLogListener.java:118)
 ~[java/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerListeners.fireListeners(ScheduledTriggers.java:806)
 ~[java/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerListeners.fireListeners(ScheduledTriggers.java:777)
 ~[java/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$add$4(ScheduledTriggers.java:289)
 ~[java/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.NodeLostTrigger.run(NodeLostTrigger.java:185) 
[java/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerWrapper.run(ScheduledTriggers.java:603)
 [java/:?]
   [junit4]   2>        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
[?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
 [?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
 [?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
[?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
[?:1.8.0_172]
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
   [junit4]   2> Caused by: java.util.concurrent.ExecutionException: 
java.io.IOException: already closed
   [junit4]   2>        at 
java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_172]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:652)
 ~[test/:?]
   [junit4]   2>        ... 13 more
   [junit4]   2> Caused by: java.io.IOException: already closed
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.sim.SimClusterStateProvider.ensureNotClosed(SimClusterStateProvider.java:2208)
 ~[test/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.sim.SimClusterStateProvider.simUpdate(SimClusterStateProvider.java:1425)
 ~[test/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.simHandleSolrRequest(SimCloudManager.java:741)
 ~[test/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.lambda$request$15(SimCloudManager.java:651)
 ~[test/:?]
   [junit4]   2>        at 
java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_172]
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
 ~[java/:?]
   [junit4]   2>        ... 3 more
   [junit4]   2> 21885 WARN  (ScheduledTrigger-162-thread-3) [    ] 
o.a.s.c.a.ScheduledTriggers Ignoring autoscaling event {
   [junit4]   2>   "id":"16ecae45b68a1T33tdptpspk9vvtgfep2dubbod",
   [junit4]   2>   "source":"node_lost_trigger2",
   [junit4]   2>   "eventTime":403292670355617,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[403292670355617],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "_enqueue_time_":403292682067467,
   [junit4]   2>     "nodeNames":["127.0.0.1:10083_solr"]}} from trigger 
org.apache.solr.cloud.autoscaling.NodeLostTrigger@161b5b6d because the executor 
has already been closed
   [junit4]   2> 21885 DEBUG (ScheduledTrigger-162-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger NodeLostTrigger processor for lost nodes: 
[127.0.0.1:10083_solr] is not ready, will try later
   [junit4]   2> 21885 DEBUG 
(TEST-TestSimTriggerIntegration.testTriggerThrottling-seed#[E16C05018E6AFDBA]) 
[    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 21885 DEBUG 
(TEST-TestSimTriggerIntegration.testTriggerThrottling-seed#[E16C05018E6AFDBA]) 
[    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed 
explicitly
   [junit4]   2> 21886 INFO  
(TEST-TestSimTriggerIntegration.testTriggerThrottling-seed#[E16C05018E6AFDBA]) 
[    ] o.a.s.SolrTestCaseJ4 ###Ending testTriggerThrottling
   [junit4] OK      0.27s J0 | TestSimTriggerIntegration.testTriggerThrottling
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.sim.TestSimTriggerIntegration_E16C05018E6AFDBA-002
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene80): {}, 
docValues:{}, maxPointsInLeafNode=558, maxMBSortInHeap=6.478728682774079, 
sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@6b853594),
 locale=es-BO, timezone=Asia/Kuala_Lumpur
   [junit4]   2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 1.8.0_172 
(64-bit)/cpus=3,threads=1,free=38448448,total=54788096
   [junit4]   2> NOTE: All tests run in this JVM: [TestSimTriggerIntegration, 
TestSimTriggerIntegration, TestSimTriggerIntegration]
   [junit4] Completed [5/5 (3!)] on J0 in 3.54s, 11 tests, 1 failure, 3 skipped 
<<< FAILURES!
   [junit4] 
   [junit4] 
   [junit4] Tests with failures [seed: E16C05018E6AFDBA]:
   [junit4]   - 
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testNodeMarkersRegistration
   [junit4]   - 
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testNodeMarkersRegistration
   [junit4]   - 
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testNodeMarkersRegistration
   [junit4] 
   [junit4] 
   [junit4] JVM J0:     0.95 ..    22.78 =    21.82s
   [junit4] JVM J1:     0.95 ..    20.63 =    19.68s
   [junit4] Execution time total: 22 seconds
   [junit4] Tests summary: 5 suites, 55 tests, 3 failures, 15 ignored (15 
assumptions)

BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:1572:
 The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:1099:
 There were test failures: 5 suites, 55 tests, 3 failures, 15 ignored (15 
assumptions) [seed: E16C05018E6AFDBA]

Total time: 25 seconds

[repro] Setting last failure code to 256

[repro] Failures:
[repro]   3/5 failed: 
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration
[repro] Exiting with code 256
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Archiving artifacts
Setting 
ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting 
ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all 
warnings.
<Git Blamer> GIT_COMMIT=1d0a08621708dde4220f9b45886a2c147d7fefd1, 
workspace=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX
[WARNINGS] Computing warning deltas based on reference build #4979
Recording test results
Setting 
ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting 
ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to