Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1747/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseParallelGC

3 tests failed.
FAILED:  
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testEventQueue

Error Message:
action wasn't interrupted

Stack Trace:
java.lang.AssertionError: action wasn't interrupted
        at 
__randomizedtesting.SeedInfo.seed([9C75F848AB27808F:55C0BAE6A240467A]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at 
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testEventQueue(TestTriggerIntegration.java:650)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testEventQueue

Error Message:
action wasn't interrupted

Stack Trace:
java.lang.AssertionError: action wasn't interrupted
        at 
__randomizedtesting.SeedInfo.seed([9C75F848AB27808F:55C0BAE6A240467A]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at 
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testEventQueue(TestTriggerIntegration.java:650)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testNodeAddedTriggerRestoreState

Error Message:
Two TriggerAction instances should have been created by now

Stack Trace:
java.lang.AssertionError: Two TriggerAction instances should have been created 
by now
        at 
__randomizedtesting.SeedInfo.seed([9C75F848AB27808F:1448713791E76122]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at 
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testNodeAddedTriggerRestoreState(TestTriggerIntegration.java:373)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 1845 lines...]
   [junit4] JVM J1: stdout was not empty, see: 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/core/test/temp/junit4-J1-20180320_211303_9537512764658171315991.sysout
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] codec: CheapBastard, pf: Memory, dvf: Direct
   [junit4] <<< JVM J1: EOF ----

[...truncated 10383 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration
   [junit4]   2> 309405 INFO  
(SUITE-TestTriggerIntegration-seed#[9C75F848AB27808F]-worker) [    ] 
o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: 
test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.sim.TestTriggerIntegration_9C75F848AB27808F-001/init-core-data-001
   [junit4]   2> 309405 WARN  
(SUITE-TestTriggerIntegration-seed#[9C75F848AB27808F]-worker) [    ] 
o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=717 numCloses=717
   [junit4]   2> 309405 INFO  
(SUITE-TestTriggerIntegration-seed#[9C75F848AB27808F]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) 
w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 309406 INFO  
(SUITE-TestTriggerIntegration-seed#[9C75F848AB27808F]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: 
@org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4] IGNOR/A 0.00s J1 | TestTriggerIntegration.testEventFromRestoredState
   [junit4]    > Assumption #1: 'badapple' test group is disabled 
(@BadApple(bugUrl=https://issues.apache.org/jira/browse/SOLR-12028))
   [junit4]   2> 309436 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[9C75F848AB27808F])
 [    ] o.a.s.SolrTestCaseJ4 ###Starting testNodeAddedTriggerRestoreState
   [junit4]   2> 309437 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[9C75F848AB27808F])
 [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and 
clearing object cache...
   [junit4]   2> 309437 DEBUG 
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[9C75F848AB27808F])
 [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed 
explicitly
   [junit4]   2> 309438 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and 
.scheduled_maintenance triggers
   [junit4]   2> 309438 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and 
.scheduled_maintenance triggers
   [junit4]   2> 309438 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 1
   [junit4]   2> 309446 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10000_solr]
   [junit4]   2> 309446 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
   [junit4]   2> 309446 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 309449 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 309449 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 309450 DEBUG (ScheduledTrigger-928-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 309471 DEBUG (ScheduledTrigger-928-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 309492 DEBUG (ScheduledTrigger-928-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 309513 DEBUG (ScheduledTrigger-928-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 309533 DEBUG (ScheduledTrigger-928-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 309553 DEBUG (ScheduledTrigger-928-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 309558 DEBUG (simCloudManagerPool-927-thread-1) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 309561 DEBUG (simCloudManagerPool-927-thread-2) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 309574 DEBUG (ScheduledTrigger-928-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 309593 DEBUG (simCloudManagerPool-927-thread-3) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 309594 DEBUG (ScheduledTrigger-928-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 309614 DEBUG (ScheduledTrigger-928-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 309636 DEBUG (ScheduledTrigger-928-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 309653 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[9C75F848AB27808F])
 [    ] o.a.s.SolrTestCaseJ4 ###Ending testNodeAddedTriggerRestoreState
   [junit4]   2> 309653 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[9C75F848AB27808F])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase 
   [junit4]   2> 
   [junit4]   2> 309653 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[9C75F848AB27808F])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase 
#############################################
   [junit4]   2> 309654 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[9C75F848AB27808F])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ############ FINAL CLUSTER STATS 
############
   [junit4]   2> 309654 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[9C75F848AB27808F])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase 
#############################################
   [junit4]   2> 
   [junit4]   2> 309654 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[9C75F848AB27808F])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Live nodes:             3
   [junit4]   2> 309654 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[9C75F848AB27808F])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Empty nodes:    3
   [junit4]   2> 309654 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[9C75F848AB27808F])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Dead nodes:             0
   [junit4]   2> 309655 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[9C75F848AB27808F])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Collections:    []
   [junit4]   2> 309655 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[9C75F848AB27808F])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Max replicas per node:  0
   [junit4]   2> 309655 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[9C75F848AB27808F])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Min replicas per node:  0
   [junit4]   2> 309655 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[9C75F848AB27808F])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Total replicas:         0
   [junit4]   2> 309655 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[9C75F848AB27808F])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Final Solr op counts 
##########
   [junit4]   2> 309656 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[9C75F848AB27808F])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ##         - autoscaling        3
   [junit4]   2> 309656 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[9C75F848AB27808F])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Autoscaling event counts 
###########
   [junit4]   2> 309656 DEBUG (ScheduledTrigger-928-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=TestTriggerIntegration 
-Dtests.method=testNodeAddedTriggerRestoreState -Dtests.seed=9C75F848AB27808F 
-Dtests.slow=true -Dtests.locale=uk-UA -Dtests.timezone=Africa/Lome 
-Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 0.22s J1 | 
TestTriggerIntegration.testNodeAddedTriggerRestoreState <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: Two TriggerAction 
instances should have been created by now
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([9C75F848AB27808F:1448713791E76122]:0)
   [junit4]    >        at 
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testNodeAddedTriggerRestoreState(TestTriggerIntegration.java:373)
   [junit4]    >        at java.lang.Thread.run(Thread.java:748)
   [junit4] IGNOR/A 0.00s J1 | 
TestTriggerIntegration.testNodeLostTriggerRestoreState
   [junit4]    > Assumption #1: 'badapple' test group is disabled 
(@BadApple(bugUrl=https://issues.apache.org/jira/browse/SOLR-12028))
   [junit4] IGNOR/A 0.00s J1 | TestTriggerIntegration.testCooldown
   [junit4]    > Assumption #1: 'badapple' test group is disabled 
(@BadApple(bugUrl=https://issues.apache.org/jira/browse/SOLR-12028))
   [junit4] IGNOR/A 0.00s J1 | 
TestTriggerIntegration.testNodeMarkersRegistration
   [junit4]    > Assumption #1: 'badapple' test group is disabled 
(@BadApple(bugUrl=https://issues.apache.org/jira/browse/SOLR-12028))
   [junit4] IGNOR/A 0.00s J1 | TestTriggerIntegration.testListeners
   [junit4]    > Assumption #1: 'badapple' test group is disabled 
(@BadApple(bugUrl=https://issues.apache.org/jira/browse/SOLR-12028))
   [junit4]   2> 309662 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.SolrTestCaseJ4 ###Starting testNodeAddedTrigger
   [junit4]   2> 309662 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing 
object cache...
   [junit4]   2> 309663 DEBUG 
(TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed 
explicitly
   [junit4]   2> 309664 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and 
.scheduled_maintenance triggers
   [junit4]   2> 309664 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 7
   [junit4]   2> 309664 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10002_solr]
   [junit4]   2> 309664 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion -1
   [junit4]   2> 309664 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
   [junit4]   2> 309664 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 309664 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
   [junit4]   2> 309664 DEBUG (ScheduledTrigger-931-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 309665 WARN  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are 
closed, exiting.
   [junit4]   2> 309685 DEBUG (ScheduledTrigger-931-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 309706 DEBUG (ScheduledTrigger-931-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 309726 DEBUG (ScheduledTrigger-931-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 309747 DEBUG (ScheduledTrigger-931-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 309764 DEBUG (simCloudManagerPool-930-thread-1) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 309765 DEBUG (simCloudManagerPool-930-thread-1) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
8
   [junit4]   2> 309765 DEBUG (simCloudManagerPool-930-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10002_solr]
   [junit4]   2> 309765 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 8
   [junit4]   2> 309766 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 309766 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 8, lastZnodeVersion 8
   [junit4]   2> 309766 DEBUG (simCloudManagerPool-930-thread-2) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 309766 DEBUG (simCloudManagerPool-930-thread-2) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
9
   [junit4]   2> 309767 DEBUG (simCloudManagerPool-930-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10002_solr]
   [junit4]   2> 309768 DEBUG (ScheduledTrigger-931-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 309769 DEBUG (simCloudManagerPool-930-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10002_solr]
   [junit4]   2> 309769 DEBUG (simCloudManagerPool-930-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated 
with properties: {event=nodeAdded, waitFor=2, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}],
 enabled=true}
   [junit4]   2> 309769 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 9
   [junit4]   2> 309769 INFO  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 309770 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion 9
   [junit4]   2> 309770 DEBUG (ScheduledTrigger-931-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 309770 DEBUG (ScheduledTrigger-931-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 309770 DEBUG (ScheduledTrigger-931-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10003_solr at time 
558733049401112
   [junit4]   2> 309788 DEBUG (ScheduledTrigger-931-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 309791 DEBUG (ScheduledTrigger-931-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 309791 DEBUG (ScheduledTrigger-931-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 309808 DEBUG (ScheduledTrigger-931-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 309814 DEBUG (ScheduledTrigger-931-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 309814 DEBUG (ScheduledTrigger-931-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 309814 DEBUG (ScheduledTrigger-931-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing 
registered processor for nodes: [127.0.0.1:10003_solr] added at times 
[558733049401112], now=558735246903112
   [junit4]   2> 309817 DEBUG (ScheduledTrigger-931-thread-3) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"1fc2a2cef0f18Tahjs4752ohvczm798cpkfh9zb",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":558733049401112,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[558733049401112],
   [junit4]   2>     "nodeNames":["127.0.0.1:10003_solr"]}}
   [junit4]   2> 309817 DEBUG (ScheduledTrigger-931-thread-3) [    ] 
o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, 
node_added_trigger]
   [junit4]   2> 309821 DEBUG (AutoscalingActionExecutor-932-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"1fc2a2cef0f18Tahjs4752ohvczm798cpkfh9zb",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":558733049401112,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[558733049401112],
   [junit4]   2>     "_enqueue_time_":558735413400112,
   [junit4]   2>     "nodeNames":["127.0.0.1:10003_solr"]}}
   [junit4]   2> 309824 DEBUG (simCloudManagerPool-930-thread-4) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 309824 DEBUG (simCloudManagerPool-930-thread-4) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
10
   [junit4]   2> 309824 DEBUG (simCloudManagerPool-930-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10002_solr, 127.0.0.1:10003_solr]
   [junit4]   2> 309824 DEBUG (simCloudManagerPool-930-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated 
with properties: {event=nodeAdded, waitFor=2, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}],
 enabled=true}
   [junit4]   2> 309824 DEBUG (simCloudManagerPool-930-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10002_solr, 127.0.0.1:10003_solr]
   [junit4]   2> 309824 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 10
   [junit4]   2> 309826 DEBUG (AutoscalingActionExecutor-932-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
   [junit4]   2> 309826 DEBUG (AutoscalingActionExecutor-932-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger after 100ms
   [junit4]   2> 309826 DEBUG (AutoscalingActionExecutor-932-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing took 243 ms for event 
id=1fc2a2cef0f18Tahjs4752ohvczm798cpkfh9zb
   [junit4]   2> 309826 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 10, lastZnodeVersion 10
   [junit4]   2> 309865 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.SolrTestCaseJ4 ###Ending testNodeAddedTrigger
   [junit4]   2> 309865 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase 
   [junit4]   2> 
   [junit4]   2> 309865 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
   [junit4]   2> 309865 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase ############ FINAL CLUSTER STATS ############
   [junit4]   2> 309865 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
   [junit4]   2> 
   [junit4]   2> 309865 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase ## Live nodes:         3
   [junit4]   2> 309865 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase ## Empty nodes:        3
   [junit4]   2> 309865 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase ## Dead nodes:         0
   [junit4]   2> 309865 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase ## Collections:        []
   [junit4]   2> 309865 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase ## Max replicas per node:      0
   [junit4]   2> 309865 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase ## Min replicas per node:      0
   [junit4]   2> 309865 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase ## Total replicas:             0
   [junit4]   2> 309865 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase ######### Final Solr op counts ##########
   [junit4]   2> 309865 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase ##             - autoscaling        3
   [junit4]   2> 309865 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase ##             - update             2
   [junit4]   2> 309865 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase ######### Autoscaling event counts 
###########
   [junit4]   2> 309866 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: node_added_trigger
   [junit4]   2> 309867 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase ##             - STARTED         1
   [junit4]   2> 309867 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase ##             - SUCCEEDED       1
   [junit4]   2> 309870 INFO  
(TEST-TestTriggerIntegration.testTriggerThrottling-seed#[9C75F848AB27808F]) [   
 ] o.a.s.SolrTestCaseJ4 ###Starting testTriggerThrottling
   [junit4]   2> 309870 DEBUG 
(TEST-TestTriggerIntegration.testTriggerThrottling-seed#[9C75F848AB27808F]) [   
 ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 11
   [junit4]   2> 309870 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 11
   [junit4]   2> 309870 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 309871 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 309871 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 11, lastZnodeVersion 11
   [junit4]   2> 309871 INFO  
(TEST-TestTriggerIntegration.testTriggerThrottling-seed#[9C75F848AB27808F]) [   
 ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and 
clearing object cache...
   [junit4]   2> 309871 WARN  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are 
closed, exiting.
   [junit4]   2> 309871 DEBUG 
(TEST-TestTriggerIntegration.testTriggerThrottling-seed#[9C75F848AB27808F]) [   
 ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed 
explicitly
   [junit4]   2> 309871 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and 
.scheduled_maintenance triggers
   [junit4]   2> 309872 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 12
   [junit4]   2> 309872 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10002_solr]
   [junit4]   2> 309872 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 12, lastZnodeVersion -1
   [junit4]   2> 309872 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 12
   [junit4]   2> 309872 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 309872 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 12, lastZnodeVersion 12
   [junit4]   2> 309872 DEBUG (ScheduledTrigger-934-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 309892 DEBUG (ScheduledTrigger-934-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 309913 DEBUG (ScheduledTrigger-934-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 309934 DEBUG (ScheduledTrigger-934-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 309954 DEBUG (ScheduledTrigger-934-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 309972 DEBUG (simCloudManagerPool-933-thread-1) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 309972 DEBUG (simCloudManagerPool-933-thread-1) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
13
   [junit4]   2> 309973 DEBUG (simCloudManagerPool-933-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10002_solr]
   [junit4]   2> 309973 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 13
   [junit4]   2> 309973 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 309973 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 13, lastZnodeVersion 13
   [junit4]   2> 309975 DEBUG (simCloudManagerPool-933-thread-2) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 309975 DEBUG (simCloudManagerPool-933-thread-2) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
14
   [junit4]   2> 309975 DEBUG (simCloudManagerPool-933-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10002_solr]
   [junit4]   2> 309975 DEBUG (simCloudManagerPool-933-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10002_solr]
   [junit4]   2> 309975 DEBUG (simCloudManagerPool-933-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated 
with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$ThrottlingTesterAction}],
 enabled=true}
   [junit4]   2> 309975 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 14
   [junit4]   2> 309975 INFO  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 309975 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 14, lastZnodeVersion 14
   [junit4]   2> 309975 DEBUG (ScheduledTrigger-934-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 309975 DEBUG (ScheduledTrigger-934-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 309976 DEBUG (ScheduledTrigger-934-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 309976 DEBUG (simCloudManagerPool-933-thread-3) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 309976 DEBUG (simCloudManagerPool-933-thread-3) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
15
   [junit4]   2> 309976 DEBUG (simCloudManagerPool-933-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10002_solr]
   [junit4]   2> 309976 DEBUG (simCloudManagerPool-933-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 instantiated 
with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$ThrottlingTesterAction}],
 enabled=true}
   [junit4]   2> 309976 DEBUG (simCloudManagerPool-933-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10002_solr]
   [junit4]   2> 309976 DEBUG (simCloudManagerPool-933-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated 
with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$ThrottlingTesterAction}],
 enabled=true}
   [junit4]   2> 309976 DEBUG (simCloudManagerPool-933-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10002_solr]
   [junit4]   2> 309976 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 15
   [junit4]   2> 309976 INFO  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 309977 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 15, lastZnodeVersion 15
   [junit4]   2> 309977 DEBUG (ScheduledTrigger-934-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 309977 DEBUG (ScheduledTrigger-934-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 309977 DEBUG (ScheduledTrigger-934-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10004_solr at time 
558743372089162
   [junit4]   2> 309977 DEBUG (ScheduledTrigger-934-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 firing 
registered processor for nodes: [127.0.0.1:10004_solr] added at times 
[558743372089162], now=558743373283612
   [junit4]   2> 309977 DEBUG (ScheduledTrigger-934-thread-3) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"1fc2c9436c74aTahjs4752ohvczm798cpkfh9ze",
   [junit4]   2>   "source":"node_added_trigger2",
   [junit4]   2>   "eventTime":558743372089162,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[558743372089162],
   [junit4]   2>     "nodeNames":["127.0.0.1:10004_solr"]}}
   [junit4]   2> 309977 DEBUG (ScheduledTrigger-934-thread-3) [    ] 
o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, 
node_added_trigger2, node_added_trigger1]
   [junit4]   2> 309978 DEBUG (AutoscalingActionExecutor-935-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"1fc2c9436c74aTahjs4752ohvczm798cpkfh9ze",
   [junit4]   2>   "source":"node_added_trigger2",
   [junit4]   2>   "eventTime":558743372089162,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[558743372089162],
   [junit4]   2>     "_enqueue_time_":558743404275912,
   [junit4]   2>     "nodeNames":["127.0.0.1:10004_solr"]}}
   [junit4]   2> 309978 INFO  (AutoscalingActionExecutor-935-thread-1) [    ] 
o.a.s.c.a.s.TestTriggerIntegration action executed from node_added_trigger2
   [junit4]   2> 309979 DEBUG (AutoscalingActionExecutor-935-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
   [junit4]   2> 309979 DEBUG (AutoscalingActionExecutor-935-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger2 after 100ms
   [junit4]   2> 309979 DEBUG (AutoscalingActionExecutor-935-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger1 after 100ms
   [junit4]   2> 309979 DEBUG (AutoscalingActionExecutor-935-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing took 51 ms for event 
id=1fc2c9436c74aTahjs4752ohvczm798cpkfh9ze
   [junit4]   2> 310079 DEBUG (ScheduledTrigger-934-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 310080 DEBUG (ScheduledTrigger-934-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 310080 DEBUG (ScheduledTrigger-934-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 310080 DEBUG (ScheduledTrigger-934-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 310080 DEBUG (ScheduledTrigger-934-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 310080 DEBUG (ScheduledTrigger-934-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10004_solr at time 
558748526416912
   [junit4]   2> 310080 DEBUG (ScheduledTrigger-934-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 firing 
registered processor for nodes: [127.0.0.1:10004_solr] added at times 
[558748526416912], now=558748528577862
   [junit4]   2> 310080 DEBUG (ScheduledTrigger-934-thread-3) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"1fc2dc76f9410Tahjs4752ohvczm798cpkfh9zh",
   [junit4]   2>   "source":"node_added_trigger1",
   [junit4]   2>   "eventTime":558748526416912,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[558748526416912],
   [junit4]   2>     "nodeNames":["127.0.0.1:10004_solr"]}}
   [junit4]   2> 310080 DEBUG (ScheduledTrigger-934-thread-3) [    ] 
o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, 
node_added_trigger2, node_added_trigger1]
   [junit4]   2> 310081 DEBUG (AutoscalingActionExecutor-935-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"1fc2dc76f9410Tahjs4752ohvczm798cpkfh9zh",
   [junit4]   2>   "source":"node_added_trigger1",
   [junit4]   2>   "eventTime":558748526416912,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[558748526416912],
   [junit4]   2>     "_enqueue_time_":558748539541512,
   [junit4]   2>     "nodeNames":["127.0.0.1:10004_solr"]}}
   [junit4]   2> 310081 INFO  (AutoscalingActionExecutor-935-thread-1) [    ] 
o.a.s.c.a.s.TestTriggerIntegration last action at 558743460367112 time = 
558748593436962
   [junit4]   2> 310081 INFO  (AutoscalingActionExecutor-935-thread-1) [    ] 
o.a.s.c.a.s.TestTriggerIntegration action executed from node_added_trigger1
   [junit4]   2> 310082 DEBUG (simCloudManagerPool-933-thread-7) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 310083 DEBUG (simCloudManagerPool-933-thread-7) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
16
   [junit4]   2> 310083 DEBUG (AutoscalingActionExecutor-935-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
   [junit4]   2> 310083 DEBUG (AutoscalingActionExecutor-935-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger2 after 100ms
   [junit4]   2> 310083 DEBUG (simCloudManagerPool-933-thread-7) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10004_solr, 127.0.0.1:10002_solr]
   [junit4]   2> 310083 DEBUG (AutoscalingActionExecutor-935-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger1 after 100ms
   [junit4]   2> 310083 DEBUG (simCloudManagerPool-933-thread-7) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 instantiated 
with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$ThrottlingTesterAction}],
 enabled=true}
   [junit4]   2> 310083 DEBUG (AutoscalingActionExecutor-935-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing took 85 ms for event 
id=1fc2dc76f9410Tahjs4752ohvczm798cpkfh9zh
   [junit4]   2> 310083 DEBUG (simCloudManagerPool-933-thread-7) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10004_solr, 127.0.0.1:10002_solr]
   [junit4]   2> 310083 DEBUG (simCloudManagerPool-933-thread-7) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10004_solr, 127.0.0.1:10002_solr]
   [junit4]   2> 310083 DEBUG (simCloudManagerPool-933-thread-7) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated 
with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$ThrottlingTesterAction}],
 enabled=true}
   [junit4]   2> 310083 DEBUG (simCloudManagerPool-933-thread-7) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10004_solr, 127.0.0.1:10002_solr]
   [junit4]   2> 310083 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 16
   [junit4]   2> 310083 INFO  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 310084 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 16, lastZnodeVersion 16
   [junit4]   2> 310084 DEBUG (ScheduledTrigger-934-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with 
currently live nodes: 3
   [junit4]   2> 310084 DEBUG (simCloudManagerPool-933-thread-9) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 310084 DEBUG (simCloudManagerPool-933-thread-9) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
17
   [junit4]   2> 310084 DEBUG (simCloudManagerPool-933-thread-9) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10004_solr, 127.0.0.1:10002_solr]
   [junit4]   2> 310084 DEBUG (simCloudManagerPool-933-thread-9) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 instantiated 
with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$ThrottlingTesterAction}],
 enabled=true}
   [junit4]   2> 310084 DEBUG (simCloudManagerPool-933-thread-9) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10004_solr, 127.0.0.1:10002_solr]
   [junit4]   2> 310084 DEBUG (simCloudManagerPool-933-thread-9) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10004_solr, 127.0.0.1:10002_solr]
   [junit4]   2> 310085 DEBUG (simCloudManagerPool-933-thread-9) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated 
with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$ThrottlingTesterAction}],
 enabled=true}
   [junit4]   2> 310085 DEBUG (simCloudManagerPool-933-thread-9) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10004_solr, 127.0.0.1:10002_solr]
   [junit4]   2> 310085 DEBUG (simCloudManagerPool-933-thread-9) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10004_solr, 127.0.0.1:10002_solr]
   [junit4]   2> 310085 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 17
   [junit4]   2> 310085 INFO  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 310085 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 17, lastZnodeVersion 17
   [junit4]   2> 310085 DEBUG (ScheduledTrigger-934-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger2 with 
currently live nodes: 3
   [junit4]   2> 310104 DEBUG (ScheduledTrigger-934-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with 
currently live nodes: 2
   [junit4]   2> 310104 DEBUG (ScheduledTrigger-934-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10004_solr
   [junit4]   2> 310104 DEBUG (ScheduledTrigger-934-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost 
nodes: [127.0.0.1:10004_solr]
   [junit4]   2> 310104 DEBUG (ScheduledTrigger-934-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"1fc2e0f676a9aTahjs4752ohvczm798cpkfh9zk",
   [junit4]   2>   "source":"node_lost_trigger1",
   [junit4]   2>   "eventTime":558749733841562,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[558749733841562],
   [junit4]   2>     "nodeNames":["127.0.0.1:10004_solr"]}}
   [junit4]   2> 310104 DEBUG (ScheduledTrigger-934-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Pausing all triggers: [node_lost_trigger2, 
.auto_add_replicas, node_added_trigger2, node_lost_trigger1, 
node_added_trigger1]
   [junit4]   2> 310105 DEBUG (AutoscalingActionExecutor-935-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"1fc2e0f676a9aTahjs4752ohvczm798cpkfh9zk",
   [junit4]   2>   "source":"node_lost_trigger1",
   [junit4]   2>   "eventTime":558749733841562,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[558749733841562],
   [junit4]   2>     "_enqueue_time_":558749754581412,
   [junit4]   2>     "nodeNames":["127.0.0.1:10004_solr"]}}
   [junit4]   2> 310106 INFO  (AutoscalingActionExecutor-935-thread-1) [    ] 
o.a.s.c.a.s.TestTriggerIntegration action executed from node_lost_trigger1
   [junit4]   2> 310106 DEBUG (AutoscalingActionExecutor-935-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: node_lost_trigger2 after 100ms
   [junit4]   2> 310106 DEBUG (AutoscalingActionExecutor-935-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
   [junit4]   2> 310106 DEBUG (AutoscalingActionExecutor-935-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger2 after 100ms
   [junit4]   2> 310106 DEBUG (AutoscalingActionExecutor-935-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: node_lost_trigger1 after 100ms
   [junit4]   2> 310106 DEBUG (AutoscalingActionExecutor-935-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger1 after 100ms
   [junit4]   2> 310106 DEBUG (AutoscalingActionExecutor-935-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing took 44 ms for event 
id=1fc2e0f676a9aTahjs4752ohvczm798cpkfh9zk
   [junit4]   2> 310207 DEBUG (ScheduledTrigger-934-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 310207 DEBUG (ScheduledTrigger-934-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 310207 DEBUG (ScheduledTrigger-934-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with 
currently live nodes: 2
   [junit4]   2> 310208 DEBUG (ScheduledTrigger-934-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 310208 DEBUG (ScheduledTrigger-934-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 310207 DEBUG (ScheduledTrigger-934-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger2 with 
currently live nodes: 2
   [junit4]   2> 310208 DEBUG (ScheduledTrigger-934-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10004_solr
   [junit4]   2> 310208 DEBUG (ScheduledTrigger-934-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost 
nodes: [127.0.0.1:10004_solr]
   [junit4]   2> 310207 DEBUG (ScheduledTrigger-934-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 310208 DEBUG (ScheduledTrigger-934-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10004_solr
   [junit4]   2> 310208 DEBUG (ScheduledTrigger-934-thread-3) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"1fc2f4420296aTahjs4752ohvczm798cpkfh9zn",
   [junit4]   2>   "source":"node_lost_trigger2",
   [junit4]   2>   "eventTime":558754913331562,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[558754913331562],
   [junit4]   2>     "nodeNames":["127.0.0.1:10004_solr"]}}
   [junit4]   2> 310208 DEBUG (ScheduledTrigger-934-thread-3) [    ] 
o.a.s.c.a.ScheduledTriggers Pausing all triggers: [node_lost_trigger2, 
.auto_add_replicas, node_added_trigger2, node_lost_trigger1, 
node_added_trigger1]
   [junit4]   2> 310209 DEBUG (AutoscalingActionExecutor-935-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"1fc2f4420296aTahjs4752ohvczm798cpkfh9zn",
   [junit4]   2>   "source":"node_lost_trigger2",
   [junit4]   2>   "eventTime":558754913331562,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[558754913331562],
   [junit4]   2>     "_enqueue_time_":558754922473612,
   [junit4]   2>     "nodeNames":["127.0.0.1:10004_solr"]}}
   [junit4]   2> 310209 INFO  (AutoscalingActionExecutor-935-thread-1) [    ] 
o.a.s.c.a.s.TestTriggerIntegration last action at 558749806817012 time = 
558754970020012
   [junit4]   2> 310209 INFO  (AutoscalingActionExecutor-935-thread-1) [    ] 
o.a.s.c.a.s.TestTriggerIntegration action executed from node_lost_trigger2
   [junit4]   2> 310209 INFO  
(TEST-TestTriggerIntegration.testTriggerThrottling-seed#[9C75F848AB27808F]) [   
 ] o.a.s.SolrTestCaseJ4 ###Ending testTriggerThrottling
   [junit4]   2> 310209 INFO  
(TEST-TestTriggerIntegration.testTriggerThrottling-seed#[9C75F848AB27808F]) [   
 ] o.a.s.c.a.s.SimSolrCloudTestCase 
   [junit4]   2> 
   [junit4]   2> 310209 INFO  
(TEST-TestTriggerIntegration.testTriggerThrottling-seed#[9C75F848AB27808F]) [   
 ] o.a.s.c.a.s.SimSolrCloudTestCase 
#############################################
   [junit4]   2> 310209 INFO  
(TEST-TestTriggerIntegration.testTriggerThrottling-seed#[9C75F848AB27808F]) [   
 ] o.a.s.c.a.s.SimSolrCloudTestCase ############ FINAL CLUSTER STATS 
############
   [junit4]   2> 310209 INFO  
(TEST-TestTriggerIntegration.testTriggerThrottling-seed#[9C75F848AB27808F]) [   
 ] o.a.s.c.a.s.SimSolrCloudTestCase 
#############################################
   [junit4]   2> 
   [junit4]   2> 310209 INFO  
(TEST-TestTriggerIntegration.testTriggerThrottling-seed#[9C75F848AB27808F]) [   
 ] o.a.s.c.a.s.SimSolrCloudTestCase ## Live nodes:                2
   [junit4]   2> 310209 INFO  
(TEST-TestTriggerIntegration.testTriggerThrottling-seed#[9C75F848AB27808F]) [   
 ] o.a.s.c.a.s.SimSolrCloudTestCase ## Empty nodes:       2
   [junit4]   2> 310209 INFO  
(TEST-TestTriggerIntegration.testTriggerThrottling-seed#[9C75F848AB27808F]) [   
 ] o.a.s.c.a.s.SimSolrCloudTestCase ## Dead nodes:                1
   [junit4]   2> 310209 INFO  
(TEST-TestTriggerIntegration.testTriggerThrottling-seed#[9C75F848AB27808F]) [   
 ] o.a.s.c.a.s.SimSolrCloudTestCase ##            127.0.0.1:10004_solr
   [junit4]   2> 310209 INFO  
(TEST-TestTriggerIntegration.testTriggerThrottling-seed#[9C75F848AB27808F]) [   
 ] o.a.s.c.a.s.SimSolrCloudTestCase ## Collections:       []
   [junit4]   2> 310209 INFO  
(TEST-TestTriggerIntegration.testTriggerThrottling-seed#[9C75F848AB27808F]) [   
 ] o.a.s.c.a.s.SimSolrCloudTestCase ## Max replicas per node:     0
   [junit4]   2> 310209 INFO  
(TEST-TestTriggerIntegration.testTriggerThrottling-seed#[9C75F848AB27808F]) [   
 ] o.a.s.c.a.s.SimSolrCloudTestCase ## Min replicas per node:     0
   [junit4]   2> 310209 INFO  
(TEST-TestTriggerIntegration.testTriggerThrottling-seed#[9C75F848AB27808F]) [   
 ] o.a.s.c.a.s.SimSolrCloudTestCase ## Total replicas:            0
   [junit4]   2> 310209 INFO  
(TEST-TestTriggerIntegration.testTriggerThrottling-seed#[9C75F848AB27808F]) [   
 ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Final Solr op counts ##########
   [junit4]   2> 310209 INFO  
(TEST-TestTriggerIntegration.testTriggerThrottling-seed#[9C75F848AB27808F]) [   
 ] o.a.s.c.a.s.SimSolrCloudTestCase ##            - autoscaling        5
   [junit4]   2> 310209 INFO  
(TEST-TestTriggerIntegration.testTriggerThrottling-seed#[9C75F848AB27808F]) [   
 ] o.a.s.c.a.s.SimSolrCloudTestCase ##            - update             7
   [junit4]   2> 310209 INFO  
(TEST-TestTriggerIntegration.testTriggerThrottling-seed#[9C75F848AB27808F]) [   
 ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Autoscaling event counts 
###########
   [junit4]   2> 310209 INFO  
(TEST-TestTriggerIntegration.testTriggerThrottling-seed#[9C75F848AB27808F]) [   
 ] o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: node_added_trigger1
   [junit4]   2> 310209 INFO  
(TEST-TestTriggerIntegration.testTriggerThrottling-seed#[9C75F848AB27808F]) [   
 ] o.a.s.c.a.s.SimSolrCloudTestCase ##            - STARTED         1
   [junit4]   2> 310209 INFO  
(TEST-TestTriggerIntegration.testTriggerThrottling-seed#[9C75F848AB27808F]) [   
 ] o.a.s.c.a.s.SimSolrCloudTestCase ##            - SUCCEEDED       1
   [junit4]   2> 310209 INFO  
(TEST-TestTriggerIntegration.testTriggerThrottling-seed#[9C75F848AB27808F]) [   
 ] o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: node_added_trigger2
   [junit4]   2> 310209 INFO  
(TEST-TestTriggerIntegration.testTriggerThrottling-seed#[9C75F848AB27808F]) [   
 ] o.a.s.c.a.s.SimSolrCloudTestCase ##            - STARTED         1
   [junit4]   2> 310209 INFO  
(TEST-TestTriggerIntegration.testTriggerThrottling-seed#[9C75F848AB27808F]) [   
 ] o.a.s.c.a.s.SimSolrCloudTestCase ##            - SUCCEEDED       1
   [junit4]   2> 310209 INFO  
(TEST-TestTriggerIntegration.testTriggerThrottling-seed#[9C75F848AB27808F]) [   
 ] o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: node_lost_trigger1
   [junit4]   2> 310209 INFO  
(TEST-TestTriggerIntegration.testTriggerThrottling-seed#[9C75F848AB27808F]) [   
 ] o.a.s.c.a.s.SimSolrCloudTestCase ##            - STARTED         1
   [junit4]   2> 310209 INFO  
(TEST-TestTriggerIntegration.testTriggerThrottling-seed#[9C75F848AB27808F]) [   
 ] o.a.s.c.a.s.SimSolrCloudTestCase ##            - SUCCEEDED       1
   [junit4]   2> 310209 DEBUG (AutoscalingActionExecutor-935-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: node_lost_trigger2 after 100ms
   [junit4]   2> 310209 INFO  
(TEST-TestTriggerIntegration.testTriggerThrottling-seed#[9C75F848AB27808F]) [   
 ] o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: node_lost_trigger2
   [junit4]   2> 310210 DEBUG (AutoscalingActionExecutor-935-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
   [junit4]   2> 310210 INFO  
(TEST-TestTriggerIntegration.testTriggerThrottling-seed#[9C75F848AB27808F]) [   
 ] o.a.s.c.a.s.SimSolrCloudTestCase ##            - STARTED         1
   [junit4]   2> 310210 DEBUG (AutoscalingActionExecutor-935-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger2 after 100ms
   [junit4]   2> 310210 DEBUG (AutoscalingActionExecutor-935-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: node_lost_trigger1 after 100ms
   [junit4]   2> 310210 DEBUG (AutoscalingActionExecutor-935-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger1 after 100ms
   [junit4]   2> 310210 DEBUG (AutoscalingActionExecutor-935-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing took 49 ms for event 
id=1fc2f4420296aTahjs4752ohvczm798cpkfh9zn
   [junit4]   2> 310215 INFO  
(TEST-TestTriggerIntegration.testEventQueue-seed#[9C75F848AB27808F]) [    ] 
o.a.s.SolrTestCaseJ4 ###Starting testEventQueue
   [junit4]   2> 310215 DEBUG 
(TEST-TestTriggerIntegration.testEventQueue-seed#[9C75F848AB27808F]) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
18
   [junit4]   2> 310215 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 18
   [junit4]   2> 310215 DEBUG 
(TEST-TestTriggerIntegration.testEventQueue-seed#[9C75F848AB27808F]) [    ] 
o.a.s.c.a.s.SimNodeStateProvider - removing dead node values: 
127.0.0.1:10004_solr
   [junit4]   2> 310215 INFO  
(TEST-TestTriggerIntegration.testEventQueue-seed#[9C75F848AB27808F]) [    ] 
o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing 
object cache...
   [junit4]   2> 310216 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 310216 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 310216 DEBUG 
(TEST-TestTriggerIntegration.testEventQueue-seed#[9C75F848AB27808F]) [    ] 
o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 310216 WARN  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Interrupted
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2>        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
   [junit4]   2>        at 
java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:165)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 310218 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and 
.scheduled_maintenance triggers
   [junit4]   2> 310218 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 19
   [junit4]   2> 310218 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10002_solr]
   [junit4]   2> 310218 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 19, lastZnodeVersion -1
   [junit4]   2> 310218 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 19
   [junit4]   2> 310220 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 310220 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 19, lastZnodeVersion 19
   [junit4]   2> 310220 DEBUG (ScheduledTrigger-937-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 310241 DEBUG (ScheduledTrigger-937-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 310263 DEBUG (ScheduledTrigger-937-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 310283 DEBUG (ScheduledTrigger-937-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 310304 DEBUG (ScheduledTrigger-937-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 310317 DEBUG (simCloudManagerPool-936-thread-1) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 310317 DEBUG (simCloudManagerPool-936-thread-1) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
20
   [junit4]   2> 310317 DEBUG (simCloudManagerPool-936-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10002_solr]
   [junit4]   2> 310318 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 20
   [junit4]   2> 310318 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 310318 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 20, lastZnodeVersion 20
   [junit4]   2> 310319 DEBUG (simCloudManagerPool-936-thread-2) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 310319 DEBUG (simCloudManagerPool-936-thread-2) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
21
   [junit4]   2> 310319 DEBUG (simCloudManagerPool-936-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10002_solr]
   [junit4]   2> 310320 INFO  (simCloudManagerPool-936-thread-2) [    ] 
o.a.s.c.a.s.TestTriggerIntegration TestEventQueueAction instantiated
   [junit4]   2> 310320 DEBUG (simCloudManagerPool-936-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10002_solr]
   [junit4]   2> 310320 DEBUG (simCloudManagerPool-936-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated 
with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestEventQueueAction}],
 enabled=true}
   [junit4]   2> 310320 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 21
   [junit4]   2> 310320 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 310320 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 21, lastZnodeVersion 21
   [junit4]   2> 310320 DEBUG (ScheduledTrigger-937-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 310320 DEBUG (ScheduledTrigger-937-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 310320 DEBUG (ScheduledTrigger-937-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10005_solr at time 
558760537553712
   [junit4]   2> 310325 DEBUG (ScheduledTrigger-937-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 310341 DEBUG (ScheduledTrigger-937-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 310341 DEBUG (ScheduledTrigger-937-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 310341 DEBUG (ScheduledTrigger-937-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 firing 
registered processor for nodes: [127.0.0.1:10005_solr] added at times 
[558760537553712], now=558761584059512
   [junit4]   2> 310341 DEBUG (ScheduledTrigger-937-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"1fc30935afb30Tahjs4752ohvczm798cpkfh9zq",
   [junit4]   2>   "source":"node_added_trigger1",
   [junit4]   2>   "eventTime":558760537553712,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[558760537553712],
   [junit4]   2>     "nodeNames":["127.0.0.1:10005_solr"]}}
   [junit4]   2> 310341 DEBUG (ScheduledTrigger-937-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, 
node_added_trigger1]
   [junit4]   2> 310342 DEBUG (AutoscalingActionExecutor-938-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"1fc30935afb30Tahjs4752ohvczm798cpkfh9zq",
   [junit4]   2>   "source":"node_added_trigger1",
   [junit4]   2>   "eventTime":558760537553712,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[558760537553712],
   [junit4]   2>     "_enqueue_time_":558761604536862,
   [junit4]   2>     "nodeNames":["127.0.0.1:10005_solr"]}}
   [junit4]   2> 310342 INFO  (AutoscalingActionExecutor-938-thread-1) [    ] 
o.a.s.c.a.s.TestTriggerIntegration -- event: {
   [junit4]   2>   "id":"1fc30935afb30Tahjs4752ohvczm798cpkfh9zq",
   [junit4]   2>   "source":"node_added_trigger1",
   [junit4]   2>   "eventTime":558760537553712,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[558760537553712],
   [junit4]   2>     "_enqueue_time_":558761604536862,
   [junit4]   2>     "nodeNames":["127.0.0.1:10005_solr"]}}
   [junit4]   2> 310342 INFO  
(TEST-TestTriggerIntegration.testEventQueue-seed#[9C75F848AB27808F]) [    ] 
o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing 
object cache...
   [junit4]   2> 310343 WARN  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are 
closed, exiting.
   [junit4]   2> 310343 DEBUG 
(TEST-TestTriggerIntegration.testEventQueue-seed#[9C75F848AB27808F]) [    ] 
o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 310343 WARN  (AutoscalingActionExecutor-938-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Exception executing actions
   [junit4]   2> org.apache.lucene.store.AlreadyClosedException: 
ScheduledTrigger node_added_trigger1 has been closed.
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerWrapper.dequeue(ScheduledTriggers.java:532)
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:317)
   [junit4]   2>        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2>        at 
java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 310343 DEBUG (AutoscalingActionExecutor-938-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing took 63 ms for event 
id=1fc30935afb30Tahjs4752ohvczm798cpkfh9zq
   [junit4]   2> 310344 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 21
   [junit4]   2> 310344 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10005_solr]
   [junit4]   2> 310344 INFO  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.s.TestTriggerIntegration TestEventQueueAction instantiated
   [junit4]   2> 310344 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10005_solr]
   [junit4]   2> 310344 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated 
with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestEventQueueAction}],
 enabled=true}
   [junit4]   2> 310344 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 21, lastZnodeVersion -1
   [junit4]   2> 310344 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 21
   [junit4]   2> 310344 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeLostTrigger Adding lost node from marker path: 
127.0.0.1:10002_solr
   [junit4]   2> 310345 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 310345 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 21, lastZnodeVersion 21
   [junit4]   2> 310345 DEBUG (ScheduledTrigger-940-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 310345 DEBUG (ScheduledTrigger-940-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10002_solr
   [junit4]   2> 310345 DEBUG (ScheduledTrigger-940-thread-2) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"1fc30935afb30Tahjs4752ohvczm798cpkfh9zq",
   [junit4]   2>   "source":"node_added_trigger1",
   [junit4]   2>   "eventTime":558760537553712,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[558760537553712],
   [junit4]   2>     "_dequeue_time_":558761773135312,
   [junit4]   2>     "_enqueue_time_":558761604536862,
   [junit4]   2>     "nodeNames":["127.0.0.1:10005_solr"],
   [junit4]   2>     "replaying":true}}
   [junit4]   2> 310345 DEBUG (ScheduledTrigger-940-thread-2) [    ] 
o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, 
node_added_trigger1]
   [junit4]   2> 310346 DEBUG (AutoscalingActionExecutor-941-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"1fc30935afb30Tahjs4752ohvczm798cpkfh9zq",
   [junit4]   2>   "source":"node_added_trigger1",
   [junit4]   2>   "eventTime":558760537553712,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[558760537553712],
   [junit4]   2>     "_dequeue_time_":558761773135312,
   [junit4]   2>     "_enqueue_time_":558761604536862,
   [junit4]   2>     "nodeNames":["127.0.0.1:10005_solr"],
   [junit4]   2>     "replaying":true}}
   [junit4]   2> 310346 INFO  (AutoscalingActionExecutor-941-thread-1) [    ] 
o.a.s.c.a.s.TestTriggerIntegration -- event: {
   [junit4]   2>   "id":"1fc30935afb30Tahjs4752ohvczm798cpkfh9zq",
   [junit4]   2>   "source":"node_added_trigger1",
   [junit4]   2>   "eventTime":558760537553712,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[558760537553712],
   [junit4]   2>     "_dequeue_time_":558761773135312,
   [junit4]   2>     "_enqueue_time_":558761604536862,
   [junit4]   2>     "nodeNames":["127.0.0.1:10005_solr"],
   [junit4]   2>     "replaying":true}}
   [junit4]   2> 310346 DEBUG (ScheduledTrigger-940-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 310346 DEBUG (ScheduledTrigger-940-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 310347 DEBUG (AutoscalingActionExecutor-941-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
   [junit4]   2> 310348 DEBUG (AutoscalingActionExecutor-941-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger1 after 100ms
   [junit4]   2> 310348 DEBUG (AutoscalingActionExecutor-941-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing took 81 ms for event 
id=1fc30935afb30Tahjs4752ohvczm798cpkfh9zq
   [junit4]   2> 310444 INFO  
(TEST-TestTriggerIntegration.testEventQueue-seed#[9C75F848AB27808F]) [    ] 
o.a.s.SolrTestCaseJ4 ###Ending testEventQueue
   [junit4]   2> 310444 INFO  
(TEST-TestTriggerIntegration.testEventQueue-seed#[9C75F848AB27808F]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase 
   [junit4]   2> 
   [junit4]   2> 310444 INFO  
(TEST-TestTriggerIntegration.testEventQueue-seed#[9C75F848AB27808F]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase #############################################
   [junit4]   2> 310444 INFO  
(TEST-TestTriggerIntegration.testEventQueue-seed#[9C75F848AB27808F]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ############ FINAL CLUSTER STATS ############
   [junit4]   2> 310444 INFO  
(TEST-TestTriggerIntegration.testEventQueue-seed#[9C75F848AB27808F]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase #############################################
   [junit4]   2> 
   [junit4]   2> 310444 INFO  
(TEST-TestTriggerIntegration.testEventQueue-seed#[9C75F848AB27808F]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ## Live nodes:               2
   [junit4]   2> 310444 INFO  
(TEST-TestTriggerIntegration.testEventQueue-seed#[9C75F848AB27808F]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ## Empty nodes:      2
   [junit4]   2> 310444 INFO  
(TEST-TestTriggerIntegration.testEventQueue-seed#[9C75F848AB27808F]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ## Dead nodes:               1
   [junit4]   2> 310444 INFO  
(TEST-TestTriggerIntegration.testEventQueue-seed#[9C75F848AB27808F]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ##           127.0.0.1:10002_solr
   [junit4]   2> 310444 INFO  
(TEST-TestTriggerIntegration.testEventQueue-seed#[9C75F848AB27808F]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ## Collections:      []
   [junit4]   2> 310444 INFO  
(TEST-TestTriggerIntegration.testEventQueue-seed#[9C75F848AB27808F]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ## Max replicas per node:    0
   [junit4]   2> 310444 INFO  
(TEST-TestTriggerIntegration.testEventQueue-seed#[9C75F848AB27808F]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ## Min replicas per node:    0
   [junit4]   2> 310444 INFO  
(TEST-TestTriggerIntegration.testEventQueue-seed#[9C75F848AB27808F]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ## Total replicas:           0
   [junit4]   2> 310444 INFO  
(TEST-TestTriggerIntegration.testEventQueue-seed#[9C75F848AB27808F]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ######### Final Solr op counts ##########
   [junit4]   2> 310444 INFO  
(TEST-TestTriggerIntegration.testEventQueue-seed#[9C75F848AB27808F]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ##           - autoscaling        2
   [junit4]   2> 310444 INFO  
(TEST-TestTriggerIntegration.testEventQueue-seed#[9C75F848AB27808F]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ##           - update             3
   [junit4]   2> 310444 INFO  
(TEST-TestTriggerIntegration.testEventQueue-seed#[9C75F848AB27808F]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ######### Autoscaling event counts ###########
   [junit4]   2> 310444 INFO  
(TEST-TestTriggerIntegration.testEventQueue-seed#[9C75F848AB27808F]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: node_added_trigger1
   [junit4]   2> 310444 INFO  
(TEST-TestTriggerIntegration.testEventQueue-seed#[9C75F848AB27808F]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ##           - STARTED         2
   [junit4]   2> 310444 INFO  
(TEST-TestTriggerIntegration.testEventQueue-seed#[9C75F848AB27808F]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ##           - SUCCEEDED       1
   [junit4] IGNOR/A 0.00s J1 | TestTriggerIntegration.testSearchRate
   [junit4]    > Assumption #1: 'badapple' test group is disabled 
(@BadApple(bugUrl=https://issues.apache.org/jira/browse/SOLR-12028))
   [junit4]   2> 310449 DEBUG (ScheduledTrigger-940-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 310449 DEBUG (ScheduledTrigger-940-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 310449 DEBUG (ScheduledTrigger-940-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 310450 INFO  
(TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.SolrTestCaseJ4 ###Starting testNodeLostTrigger
   [junit4]   2> 310450 DEBUG 
(TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 22
   [junit4]   2> 310450 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 22
   [junit4]   2> 310450 DEBUG 
(TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimNodeStateProvider - removing dead node values: 
127.0.0.1:10002_solr
   [junit4]   2> 310450 INFO  
(TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing 
object cache...
   [junit4]   2> 310450 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 310450 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 310450 WARN  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Interrupted
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2>        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
   [junit4]   2>        at 
java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:165)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 310451 DEBUG 
(TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed 
explicitly
   [junit4]   2> 310451 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and 
.scheduled_maintenance triggers
   [junit4]   2> 310451 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 23
   [junit4]   2> 310451 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10005_solr]
   [junit4]   2> 310451 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 23, lastZnodeVersion -1
   [junit4]   2> 310451 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 23
   [junit4]   2> 310452 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 310452 DEBUG (Sim

[...truncated too long message...]

_lost_trigger with currently live nodes: 1
   [junit4]   2> 15599 DEBUG (ScheduledTrigger-146-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost 
nodes: [127.0.0.1:10044_solr]
   [junit4]   2> 15599 DEBUG (ScheduledTrigger-146-thread-2) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"20123daee2868Tahjs4752ohvczm798cpkfh8a4",
   [junit4]   2>   "source":"node_lost_trigger",
   [junit4]   2>   "eventTime":564203461945448,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[564203461945448],
   [junit4]   2>     "nodeNames":["127.0.0.1:10044_solr"]}}
   [junit4]   2> 15599 DEBUG (ScheduledTrigger-146-thread-2) [    ] 
o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, 
node_lost_trigger]
   [junit4]   2> 15600 DEBUG (AutoscalingActionExecutor-147-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"20123daee2868Tahjs4752ohvczm798cpkfh8a4",
   [junit4]   2>   "source":"node_lost_trigger",
   [junit4]   2>   "eventTime":564203461945448,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[564203461945448],
   [junit4]   2>     "_enqueue_time_":564207280718898,
   [junit4]   2>     "nodeNames":["127.0.0.1:10044_solr"]}}
   [junit4]   2> 15601 DEBUG (simCloudManagerPool-145-thread-6) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 15601 DEBUG (simCloudManagerPool-145-thread-6) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
56
   [junit4]   2> 15601 DEBUG (simCloudManagerPool-145-thread-6) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10039_solr]
   [junit4]   2> 15601 DEBUG (simCloudManagerPool-145-thread-6) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10039_solr]
   [junit4]   2> 15601 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 56
   [junit4]   2> 15602 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 15602 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 56, lastZnodeVersion 56
   [junit4]   2> 15603 DEBUG (AutoscalingActionExecutor-147-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
   [junit4]   2> 15603 DEBUG (AutoscalingActionExecutor-147-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: node_lost_trigger after 100ms
   [junit4]   2> 15603 DEBUG (AutoscalingActionExecutor-147-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing took 165 ms for event 
id=20123daee2868Tahjs4752ohvczm798cpkfh8a4
   [junit4]   2> 15645 INFO  
(TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.SolrTestCaseJ4 ###Ending testNodeLostTrigger
   [junit4]   2> 15645 INFO  
(TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase 
   [junit4]   2> 
   [junit4]   2> 15645 INFO  
(TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
   [junit4]   2> 15645 INFO  
(TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase ############ FINAL CLUSTER STATS ############
   [junit4]   2> 15645 INFO  
(TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
   [junit4]   2> 
   [junit4]   2> 15645 INFO  
(TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase ## Live nodes:           1
   [junit4]   2> 15645 INFO  
(TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase ## Empty nodes:  1
   [junit4]   2> 15645 INFO  
(TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase ## Dead nodes:           1
   [junit4]   2> 15645 INFO  
(TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase ##               127.0.0.1:10044_solr
   [junit4]   2> 15645 INFO  
(TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase ## Collections:  []
   [junit4]   2> 15645 INFO  
(TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase ## Max replicas per node:        0
   [junit4]   2> 15645 INFO  
(TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase ## Min replicas per node:        0
   [junit4]   2> 15645 INFO  
(TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase ## Total replicas:               0
   [junit4]   2> 15645 INFO  
(TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase ######### Final Solr op counts ##########
   [junit4]   2> 15646 INFO  
(TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase ##               - autoscaling        3
   [junit4]   2> 15646 INFO  
(TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase ##               - update             2
   [junit4]   2> 15646 INFO  
(TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase ######### Autoscaling event counts 
###########
   [junit4]   2> 15646 INFO  
(TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: node_lost_trigger
   [junit4]   2> 15646 INFO  
(TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase ##               - STARTED         1
   [junit4]   2> 15646 INFO  
(TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase ##               - SUCCEEDED       1
   [junit4]   2> 15646 INFO  
(TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: search_rate_trigger
   [junit4]   2> 15646 INFO  
(TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[9C75F848AB27808F]) [    
] o.a.s.c.a.s.SimSolrCloudTestCase ##               - FAILED          1
   [junit4] OK      0.30s J0 | TestTriggerIntegration.testNodeLostTrigger
   [junit4]   2> 15646 DEBUG 
(SUITE-TestTriggerIntegration-seed#[9C75F848AB27808F]-worker) [    ] 
o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 15647 WARN  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are 
closed, exiting.
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.sim.TestTriggerIntegration_9C75F848AB27808F-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, 
docValues:{}, maxPointsInLeafNode=219, maxMBSortInHeap=5.086037509597258, 
sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@6680b77b),
 locale=uk-UA, timezone=Africa/Lome
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_162 
(64-bit)/cpus=3,threads=1,free=199766216,total=218103808
   [junit4]   2> NOTE: All tests run in this JVM: [TestTriggerIntegration, 
TestTriggerIntegration, TestTriggerIntegration]
   [junit4] Completed [5/5 (2!)] on J0 in 4.19s, 11 tests, 1 failure <<< 
FAILURES!

[...truncated 11 lines...]
BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:1561:
 The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:1085:
 There were test failures: 5 suites, 55 tests, 2 failures [seed: 
9C75F848AB27808F]

Total time: 28 seconds

[repro] Setting last failure code to 256

[repro] Failures:
[repro]   2/5 failed: 
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration
[repro] Exiting with code 256
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Archiving artifacts
Setting 
ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting 
ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all 
warnings.
<Git Blamer> GIT_COMMIT=db62ffe94ac734c241942822bc5e26dae4192b23, 
workspace=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris
[WARNINGS] Computing warning deltas based on reference build #1746
Recording test results
Setting 
ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting 
ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to