Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/291/

3 tests failed.
FAILED:  
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testEventFromRestoredState

Error Message:


Stack Trace:
java.lang.AssertionError
        at 
__randomizedtesting.SeedInfo.seed([1F4E1C5F6521068F:1F78A89F4C2CA2E5]:0)
        at org.junit.Assert.fail(Assert.java:92)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertTrue(Assert.java:54)
        at 
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testEventFromRestoredState(TestTriggerIntegration.java:674)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testTriggerThrottling

Error Message:
Two TriggerAction instances should have been created by now

Stack Trace:
java.lang.AssertionError: Two TriggerAction instances should have been created 
by now
        at 
__randomizedtesting.SeedInfo.seed([1F4E1C5F6521068F:E46CB47AB78BE51D]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at 
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testTriggerThrottling(TestTriggerIntegration.java:205)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testSearchRate

Error Message:
The trigger did not fire at all

Stack Trace:
java.lang.AssertionError: The trigger did not fire at all
        at 
__randomizedtesting.SeedInfo.seed([1F4E1C5F6521068F:420602D6AAE7A0C0]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at 
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testSearchRate(TestTriggerIntegration.java:1178)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 12182 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration
   [junit4]   2> Creating dataDir: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.sim.TestTriggerIntegration_1F4E1C5F6521068F-001/init-core-data-001
   [junit4]   2> 1237361 WARN  
(SUITE-TestTriggerIntegration-seed#[1F4E1C5F6521068F]-worker) [    ] 
o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=4 numCloses=4
   [junit4]   2> 1237361 INFO  
(SUITE-TestTriggerIntegration-seed#[1F4E1C5F6521068F]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) 
w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 1237415 INFO  
(SUITE-TestTriggerIntegration-seed#[1F4E1C5F6521068F]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: 
@org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 1237416 INFO  
(SUITE-TestTriggerIntegration-seed#[1F4E1C5F6521068F]-worker) [    ] 
o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: 
test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 1237432 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 1237432 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 0
   [junit4]   2> 1237432 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10008_solr, 
127.0.0.1:10007_solr]
   [junit4]   2> 1237432 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion -1
   [junit4]   2> 1237432 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 0
   [junit4]   2> 1237452 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1237452 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion 0
   [junit4]   2> 1237452 DEBUG (ScheduledTrigger-1485-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1237562 DEBUG (ScheduledTrigger-1485-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1237565 INFO  
(TEST-TestTriggerIntegration.testListeners-seed#[1F4E1C5F6521068F]) [    ] 
o.a.s.SolrTestCaseJ4 ###Starting testListeners
   [junit4]   2> 1237566 DEBUG 
(TEST-TestTriggerIntegration.testListeners-seed#[1F4E1C5F6521068F]) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
1
   [junit4]   2> 1237566 INFO  
(TEST-TestTriggerIntegration.testListeners-seed#[1F4E1C5F6521068F]) [    ] 
o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing 
object cache...
   [junit4]   2> 1237578 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 1237598 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1237598 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1237599 WARN  (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Interrupted
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2>        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
   [junit4]   2>        at 
java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:154)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1237599 DEBUG 
(TEST-TestTriggerIntegration.testListeners-seed#[1F4E1C5F6521068F]) [    ] 
o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 1237612 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 1237627 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 2
   [junit4]   2> 1237627 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10008_solr, 
127.0.0.1:10007_solr]
   [junit4]   2> 1237627 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion -1
   [junit4]   2> 1237627 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 1237647 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1237647 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
   [junit4]   2> 1237661 DEBUG (simCloudManagerPool-1484-thread-1) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1237662 DEBUG (simCloudManagerPool-1484-thread-1) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
3
   [junit4]   2> 1237691 DEBUG (ScheduledTrigger-1487-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1237711 DEBUG (ScheduledTrigger-1487-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1237712 DEBUG (simCloudManagerPool-1484-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10008_solr, 
127.0.0.1:10007_solr]
   [junit4]   2> 1237715 DEBUG (simCloudManagerPool-1484-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10008_solr, 
127.0.0.1:10007_solr]
   [junit4]   2> 1237715 DEBUG (simCloudManagerPool-1484-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated 
with properties: {event=nodeAdded, waitFor=2, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction},
 {name=test1, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestDummyAction}],
 enabled=true}
   [junit4]   2> 1237736 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 1237737 INFO  (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 1237756 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 1237769 DEBUG (ScheduledTrigger-1487-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1237806 DEBUG (ScheduledTrigger-1487-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 1237794 DEBUG (simCloudManagerPool-1484-thread-2) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1237810 DEBUG (simCloudManagerPool-1484-thread-2) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
4
   [junit4]   2> 1237810 DEBUG (simCloudManagerPool-1484-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10008_solr, 
127.0.0.1:10007_solr]
   [junit4]   2> 1237810 DEBUG (simCloudManagerPool-1484-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10008_solr, 
127.0.0.1:10007_solr]
   [junit4]   2> 1237810 DEBUG (simCloudManagerPool-1484-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated 
with properties: {event=nodeAdded, waitFor=2, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction},
 {name=test1, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestDummyAction}],
 enabled=true}
   [junit4]   2> 1237810 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 1237810 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
   [junit4]   2> 1237909 DEBUG (ScheduledTrigger-1487-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1237909 DEBUG (ScheduledTrigger-1487-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1237909 DEBUG (ScheduledTrigger-1487-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 1237924 DEBUG (simCloudManagerPool-1484-thread-3) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1237924 DEBUG (simCloudManagerPool-1484-thread-3) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
5
   [junit4]   2> 1237924 DEBUG (simCloudManagerPool-1484-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10008_solr, 
127.0.0.1:10007_solr]
   [junit4]   2> 1237924 DEBUG (simCloudManagerPool-1484-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10008_solr, 
127.0.0.1:10007_solr]
   [junit4]   2> 1237924 DEBUG (simCloudManagerPool-1484-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated 
with properties: {event=nodeAdded, waitFor=2, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction},
 {name=test1, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestDummyAction}],
 enabled=true}
   [junit4]   2> 1237931 DEBUG (ScheduledTrigger-1487-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 1237932 DEBUG (ScheduledTrigger-1487-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1237932 DEBUG (ScheduledTrigger-1487-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1237932 DEBUG (ScheduledTrigger-1487-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10009_solr at time 
14046340574959915
   [junit4]   2> 1237938 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
   [junit4]   2> 1237938 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
   [junit4]   2> 1237952 DEBUG (ScheduledTrigger-1487-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 1237952 DEBUG (ScheduledTrigger-1487-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1237952 DEBUG (ScheduledTrigger-1487-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1237980 DEBUG (ScheduledTrigger-1487-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 1237980 DEBUG (ScheduledTrigger-1487-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1237980 DEBUG (ScheduledTrigger-1487-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1237981 DEBUG (ScheduledTrigger-1487-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing 
registered processor for nodes: [127.0.0.1:10009_solr] added at times 
[14046340574959915], now=14046343020226115
   [junit4]   2> 1237981 DEBUG (ScheduledTrigger-1487-thread-4) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"31e712845d592bTd2gueqt0mxaatvirnw4stveon",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":14046340574959915,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[14046340574959915],
   [junit4]   2>     "nodeNames":["127.0.0.1:10009_solr"]}}
   [junit4]   2> 1237984 DEBUG (AutoscalingActionExecutor-1488-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"31e712845d592bTd2gueqt0mxaatvirnw4stveon",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":14046340574959915,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[14046340574959915],
   [junit4]   2>     "_enqueue_time_":14046343037851765,
   [junit4]   2>     "nodeNames":["127.0.0.1:10009_solr"]}}
   [junit4]   2> 1237999 DEBUG (AutoscalingActionExecutor-1488-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing took 723 ms for event 
id=31e712845d592bTd2gueqt0mxaatvirnw4stveon
   [junit4]   2> 1238001 DEBUG (ScheduledTrigger-1487-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 1238004 DEBUG (ScheduledTrigger-1487-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1238004 DEBUG (ScheduledTrigger-1487-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1238021 DEBUG (ScheduledTrigger-1487-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 1238024 DEBUG (ScheduledTrigger-1487-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1238024 DEBUG (ScheduledTrigger-1487-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1238041 DEBUG (ScheduledTrigger-1487-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 1238045 DEBUG (ScheduledTrigger-1487-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1238045 DEBUG (ScheduledTrigger-1487-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1238070 DEBUG (ScheduledTrigger-1487-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4
   [junit4]   2> 1238070 DEBUG (ScheduledTrigger-1487-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1238070 DEBUG (ScheduledTrigger-1487-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 1238070 DEBUG (ScheduledTrigger-1487-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10010_solr at time 
14046347489671215
   [junit4]   2> 1238090 DEBUG (ScheduledTrigger-1487-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4
   [junit4]   2> 1238091 DEBUG (ScheduledTrigger-1487-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1238091 DEBUG (ScheduledTrigger-1487-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 1238110 DEBUG (ScheduledTrigger-1487-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4
   [junit4]   2> 1238119 DEBUG (ScheduledTrigger-1487-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1238119 DEBUG (ScheduledTrigger-1487-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 1238119 DEBUG (ScheduledTrigger-1487-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing 
registered processor for nodes: [127.0.0.1:10010_solr] added at times 
[14046347489671215], now=14046349925255115
   [junit4]   2> 1238119 DEBUG (ScheduledTrigger-1487-thread-4) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"31e7142083782fTd2gueqt0mxaatvirnw4stveoq",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":14046347489671215,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[14046347489671215],
   [junit4]   2>     "nodeNames":["127.0.0.1:10010_solr"]}}
   [junit4]   2> 1238120 DEBUG (AutoscalingActionExecutor-1488-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"31e7142083782fTd2gueqt0mxaatvirnw4stveoq",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":14046347489671215,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[14046347489671215],
   [junit4]   2>     "_enqueue_time_":14046349942536665,
   [junit4]   2>     "nodeNames":["127.0.0.1:10010_solr"]}}
   [junit4]   2> 1238120 INFO  (AutoscalingActionExecutor-1488-thread-1) [    ] 
o.a.s.c.ActionThrottle The last action attempt started 6147ms ago.
   [junit4]   2> 1238121 WARN  (AutoscalingActionExecutor-1488-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Exception executing actions
   [junit4]   2> java.lang.Exception: Error executing action: test1 for trigger 
event: {
   [junit4]   2>   "id":"31e7142083782fTd2gueqt0mxaatvirnw4stveoq",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":14046347489671215,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[14046347489671215],
   [junit4]   2>     "_enqueue_time_":14046349942536665,
   [junit4]   2>     "nodeNames":["127.0.0.1:10010_solr"]}}
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:330)
   [junit4]   2>        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2>        at 
java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: java.lang.RuntimeException: failure
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestDummyAction.process(TestTriggerIntegration.java:890)
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:327)
   [junit4]   2>        ... 6 more
   [junit4]   2> 1238121 DEBUG (AutoscalingActionExecutor-1488-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing took 95 ms for event 
id=31e7142083782fTd2gueqt0mxaatvirnw4stveoq
   [junit4]   2> 1238135 DEBUG (ScheduledTrigger-1487-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4
   [junit4]   2> 1238140 DEBUG (ScheduledTrigger-1487-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1238140 DEBUG (ScheduledTrigger-1487-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 1238155 DEBUG (ScheduledTrigger-1487-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4
   [junit4]   2> 1238163 DEBUG (ScheduledTrigger-1487-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1238163 DEBUG (ScheduledTrigger-1487-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 1238164 INFO  
(TEST-TestTriggerIntegration.testListeners-seed#[1F4E1C5F6521068F]) [    ] 
o.a.s.SolrTestCaseJ4 ###Ending testListeners
   [junit4]   2> 1238167 INFO  
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[1F4E1C5F6521068F])
 [    ] o.a.s.SolrTestCaseJ4 ###Starting testNodeMarkersRegistration
   [junit4]   2> 1238167 DEBUG 
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[1F4E1C5F6521068F])
 [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 6
   [junit4]   2> 1238167 INFO  
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[1F4E1C5F6521068F])
 [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and 
clearing object cache...
   [junit4]   2> 1238167 WARN  (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are 
closed, exiting.
   [junit4]   2> 1238167 DEBUG 
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[1F4E1C5F6521068F])
 [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed 
explicitly
   [junit4]   2> 1238237 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 1238237 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 7
   [junit4]   2> 1238237 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 
127.0.0.1:10009_solr, 127.0.0.1:10008_solr, 127.0.0.1:10007_solr, 
127.0.0.1:10011_solr]
   [junit4]   2> 1238237 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion -1
   [junit4]   2> 1238237 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
   [junit4]   2> 1238200 INFO  
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[1F4E1C5F6521068F])
 [    ] o.a.s.c.a.s.TestTriggerIntegration ====== KILL OVERSEER 1
   [junit4]   2> 1238237 INFO  
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[1F4E1C5F6521068F])
 [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and 
clearing object cache...
   [junit4]   2> 1238257 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1238257 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
   [junit4]   2> 1238294 DEBUG 
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[1F4E1C5F6521068F])
 [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed 
explicitly
   [junit4]   2> 1238302 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 7
   [junit4]   2> 1238302 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 
127.0.0.1:10008_solr, 127.0.0.1:10007_solr, 127.0.0.1:10011_solr]
   [junit4]   2> 1238302 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion -1
   [junit4]   2> 1238302 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
   [junit4]   2> 1238302 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.NodeLostTrigger Adding lost node from marker path: 
127.0.0.1:10009_solr
   [junit4]   2> 1238302 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1238302 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
   [junit4]   2> 1238302 DEBUG (ScheduledTrigger-1491-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4
   [junit4]   2> 1238323 DEBUG (ScheduledTrigger-1491-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4
   [junit4]   2> 1238346 WARN  (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are 
closed, exiting.
   [junit4]   2> 1238379 DEBUG (ScheduledTrigger-1491-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4
   [junit4]   2> 1238408 DEBUG (ScheduledTrigger-1491-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4
   [junit4]   2> 1238411 INFO  
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[1F4E1C5F6521068F])
 [    ] o.a.s.c.a.s.TestTriggerIntegration ====== ADD TRIGGERS
   [junit4]   2> 1238436 DEBUG (ScheduledTrigger-1491-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4
   [junit4]   2> 1238450 DEBUG (simCloudManagerPool-1484-thread-8) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1238450 DEBUG (simCloudManagerPool-1484-thread-8) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
8
   [junit4]   2> 1238450 DEBUG (simCloudManagerPool-1484-thread-8) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 
127.0.0.1:10008_solr, 127.0.0.1:10007_solr, 127.0.0.1:10011_solr]
   [junit4]   2> 1238450 DEBUG (simCloudManagerPool-1484-thread-8) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10010_solr, 
127.0.0.1:10008_solr, 127.0.0.1:10007_solr, 127.0.0.1:10011_solr]
   [junit4]   2> 1238450 DEBUG (simCloudManagerPool-1484-thread-8) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated 
with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestEventMarkerAction}],
 enabled=true}
   [junit4]   2> 1238451 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 8
   [junit4]   2> 1238451 INFO  (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.s.TestTriggerIntegration TestEventMarkerAction init
   [junit4]   2> 1238456 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 8, lastZnodeVersion 8
   [junit4]   2> 1238456 DEBUG (ScheduledTrigger-1491-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1238456 DEBUG (ScheduledTrigger-1491-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 1238456 DEBUG (ScheduledTrigger-1491-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4
   [junit4]   2> 1238457 DEBUG (simCloudManagerPool-1484-thread-9) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1238457 DEBUG (simCloudManagerPool-1484-thread-9) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
9
   [junit4]   2> 1238457 DEBUG (simCloudManagerPool-1484-thread-9) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 
127.0.0.1:10008_solr, 127.0.0.1:10007_solr, 127.0.0.1:10011_solr]
   [junit4]   2> 1238457 DEBUG (simCloudManagerPool-1484-thread-9) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 
127.0.0.1:10008_solr, 127.0.0.1:10007_solr, 127.0.0.1:10011_solr]
   [junit4]   2> 1238457 DEBUG (simCloudManagerPool-1484-thread-9) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10010_solr, 
127.0.0.1:10008_solr, 127.0.0.1:10007_solr, 127.0.0.1:10011_solr]
   [junit4]   2> 1238457 DEBUG (simCloudManagerPool-1484-thread-9) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated 
with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestEventMarkerAction}],
 enabled=true}
   [junit4]   2> 1238457 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 9
   [junit4]   2> 1238458 INFO  (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.s.TestTriggerIntegration TestEventMarkerAction init
   [junit4]   2> 1238458 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion 9
   [junit4]   2> 1238458 DEBUG (ScheduledTrigger-1491-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 4
   [junit4]   2> 1238458 INFO  
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[1F4E1C5F6521068F])
 [    ] o.a.s.c.a.s.TestTriggerIntegration ====== ADD NODE 1
   [junit4]   2> 1238477 DEBUG (ScheduledTrigger-1491-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 5
   [junit4]   2> 1238497 DEBUG (ScheduledTrigger-1491-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 5
   [junit4]   2> 1238497 DEBUG (ScheduledTrigger-1491-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1238497 DEBUG (ScheduledTrigger-1491-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 5
   [junit4]   2> 1238497 DEBUG (ScheduledTrigger-1491-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10012_solr at time 
14046368854241865
   [junit4]   2> 1238497 DEBUG (ScheduledTrigger-1491-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 5
   [junit4]   2> 1238521 DEBUG (ScheduledTrigger-1491-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 5
   [junit4]   2> 1238521 DEBUG (ScheduledTrigger-1491-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1238521 DEBUG (ScheduledTrigger-1491-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 5
   [junit4]   2> 1238521 DEBUG (ScheduledTrigger-1491-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing 
registered processor for nodes: [127.0.0.1:10012_solr] added at times 
[14046368854241865], now=14046370065150315
   [junit4]   2> 1238522 DEBUG (ScheduledTrigger-1491-thread-2) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"31e71919f0f249Td2gueqt0mxaatvirnw4stveot",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":14046368854241865,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[14046368854241865],
   [junit4]   2>     "nodeNames":["127.0.0.1:10012_solr"]}}
   [junit4]   2> 1238550 DEBUG (AutoscalingActionExecutor-1492-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"31e71919f0f249Td2gueqt0mxaatvirnw4stveot",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":14046368854241865,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[14046368854241865],
   [junit4]   2>     "_enqueue_time_":14046370072147165,
   [junit4]   2>     "nodeNames":["127.0.0.1:10012_solr"]}}
   [junit4]   2> 1238566 DEBUG (AutoscalingActionExecutor-1492-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing took 810 ms for event 
id=31e71919f0f249Td2gueqt0mxaatvirnw4stveot
   [junit4]   2> 1238574 DEBUG (ScheduledTrigger-1491-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 5
   [junit4]   2> 1238574 DEBUG (ScheduledTrigger-1491-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1238575 DEBUG (ScheduledTrigger-1491-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 5
   [junit4]   2> 1238575 DEBUG (ScheduledTrigger-1491-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 5
   [junit4]   2> 1238575 INFO  
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[1F4E1C5F6521068F])
 [    ] o.a.s.c.a.s.TestTriggerIntegration ====== KILL OVERSEER 2
   [junit4]   2> 1238575 INFO  
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[1F4E1C5F6521068F])
 [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and 
clearing object cache...
   [junit4]   2> 1238575 WARN  (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are 
closed, exiting.
   [junit4]   2> 1238575 DEBUG 
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[1F4E1C5F6521068F])
 [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed 
explicitly
   [junit4]   2> 1238598 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 9
   [junit4]   2> 1238598 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 
127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 127.0.0.1:10007_solr]
   [junit4]   2> 1238598 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 
127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 127.0.0.1:10007_solr]
   [junit4]   2> 1238611 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10010_solr, 
127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 127.0.0.1:10007_solr]
   [junit4]   2> 1238611 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated 
with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestEventMarkerAction}],
 enabled=true}
   [junit4]   2> 1238611 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion -1
   [junit4]   2> 1238611 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 9
   [junit4]   2> 1238613 INFO  (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.s.TestTriggerIntegration TestEventMarkerAction init
   [junit4]   2> 1238613 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.NodeLostTrigger Adding lost node from marker path: 
127.0.0.1:10011_solr
   [junit4]   2> 1238615 INFO  (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.s.TestTriggerIntegration TestEventMarkerAction init
   [junit4]   2> 1238623 DEBUG (ScheduledTrigger-1493-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 4
   [junit4]   2> 1238623 DEBUG (ScheduledTrigger-1493-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10011_solr
   [junit4]   2> 1238632 DEBUG (ScheduledTrigger-1493-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1238632 DEBUG (ScheduledTrigger-1493-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 1238632 DEBUG (ScheduledTrigger-1493-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4
   [junit4]   2> 1238632 DEBUG (ScheduledTrigger-1493-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10011_solr
   [junit4]   2> 1238632 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion 9
   [junit4]   2> 1238652 DEBUG (ScheduledTrigger-1493-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 4
   [junit4]   2> 1238652 DEBUG (ScheduledTrigger-1493-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost 
nodes: [127.0.0.1:10011_solr]
   [junit4]   2> 1238652 DEBUG (ScheduledTrigger-1493-thread-2) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"31e71a91ae99d9Td2gueqt0mxaatvirnw4stveow",
   [junit4]   2>   "source":"node_lost_trigger",
   [junit4]   2>   "eventTime":14046375158127065,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[14046375158127065],
   [junit4]   2>     "nodeNames":["127.0.0.1:10011_solr"]}}
   [junit4]   2> 1238653 DEBUG (ScheduledTrigger-1493-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1238653 DEBUG (ScheduledTrigger-1493-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 1238689 DEBUG (ScheduledTrigger-1493-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4
   [junit4]   2> 1238698 DEBUG (AutoscalingActionExecutor-1494-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"31e71a91ae99d9Td2gueqt0mxaatvirnw4stveow",
   [junit4]   2>   "source":"node_lost_trigger",
   [junit4]   2>   "eventTime":14046375158127065,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[14046375158127065],
   [junit4]   2>     "_enqueue_time_":14046376611006715,
   [junit4]   2>     "nodeNames":["127.0.0.1:10011_solr"]}}
   [junit4]   2> 1238699 INFO  
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[1F4E1C5F6521068F])
 [    ] o.a.s.SolrTestCaseJ4 ###Ending testNodeMarkersRegistration
   [junit4]   2> 1238742 DEBUG (AutoscalingActionExecutor-1494-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing took 2185 ms for event 
id=31e71a91ae99d9Td2gueqt0mxaatvirnw4stveow
   [junit4]   2> 1238744 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[1F4E1C5F6521068F]) [    
] o.a.s.SolrTestCaseJ4 ###Starting testNodeAddedTrigger
   [junit4]   2> 1238744 DEBUG 
(TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[1F4E1C5F6521068F]) [    
] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 10
   [junit4]   2> 1238744 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 
10
   [junit4]   2> 1238744 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[1F4E1C5F6521068F]) [    
] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing 
object cache...
   [junit4]   2> 1238745 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1238745 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1238745 DEBUG 
(TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[1F4E1C5F6521068F]) [    
] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed 
explicitly
   [junit4]   2> 1238745 WARN  (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Interrupted
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2>        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
   [junit4]   2>        at 
java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:154)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1238778 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 1238778 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 11
   [junit4]   2> 1238779 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 
127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 127.0.0.1:10007_solr]
   [junit4]   2> 1238779 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 11, lastZnodeVersion -1
   [junit4]   2> 1238779 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 
11
   [junit4]   2> 1238779 DEBUG (simCloudManagerPool-1484-thread-14) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1238779 DEBUG (simCloudManagerPool-1484-thread-14) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
12
   [junit4]   2> 1238784 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1238811 DEBUG (simCloudManagerPool-1484-thread-14) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 
127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 127.0.0.1:10007_solr]
   [junit4]   2> 1238811 DEBUG (simCloudManagerPool-1484-thread-14) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10010_solr, 
127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 127.0.0.1:10007_solr]
   [junit4]   2> 1238811 DEBUG (simCloudManagerPool-1484-thread-14) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated 
with properties: {event=nodeAdded, waitFor=3, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}],
 enabled=true}
   [junit4]   2> 1238811 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 12, lastZnodeVersion 11
   [junit4]   2> 1238811 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 
12
   [junit4]   2> 1238811 INFO  (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 1238827 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 12, lastZnodeVersion 12
   [junit4]   2> 1238827 DEBUG (ScheduledTrigger-1495-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 5
   [junit4]   2> 1238827 DEBUG (ScheduledTrigger-1495-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1238827 DEBUG (ScheduledTrigger-1495-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 5
   [junit4]   2> 1238827 DEBUG (ScheduledTrigger-1495-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10013_solr at time 
14046385339987865
   [junit4]   2> 1238852 DEBUG (ScheduledTrigger-1495-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 5
   [junit4]   2> 1238852 DEBUG (ScheduledTrigger-1495-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1238852 DEBUG (ScheduledTrigger-1495-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 5
   [junit4]   2> 1238884 DEBUG (ScheduledTrigger-1495-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 5
   [junit4]   2> 1238884 DEBUG (ScheduledTrigger-1495-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1238884 DEBUG (ScheduledTrigger-1495-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 5
   [junit4]   2> 1238904 DEBUG (ScheduledTrigger-1495-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 5
   [junit4]   2> 1238904 DEBUG (ScheduledTrigger-1495-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1238904 DEBUG (ScheduledTrigger-1495-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 5
   [junit4]   2> 1238904 DEBUG (ScheduledTrigger-1495-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing 
registered processor for nodes: [127.0.0.1:10013_solr] added at times 
[14046385339987865], now=14046389198907365
   [junit4]   2> 1238904 DEBUG (ScheduledTrigger-1495-thread-3) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"31e71cf0917799Td2gueqt0mxaatvirnw4stveoz",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":14046385339987865,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[14046385339987865],
   [junit4]   2>     "nodeNames":["127.0.0.1:10013_solr"]}}
   [junit4]   2> 1238906 DEBUG (AutoscalingActionExecutor-1496-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"31e71cf0917799Td2gueqt0mxaatvirnw4stveoz",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":14046385339987865,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[14046385339987865],
   [junit4]   2>     "_enqueue_time_":14046389209028065,
   [junit4]   2>     "nodeNames":["127.0.0.1:10013_solr"]}}
   [junit4]   2> 1238930 DEBUG (AutoscalingActionExecutor-1496-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing took 1212 ms for event 
id=31e71cf0917799Td2gueqt0mxaatvirnw4stveoz
   [junit4]   2> 1238935 DEBUG (simCloudManagerPool-1484-thread-17) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1238935 DEBUG (simCloudManagerPool-1484-thread-17) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
13
   [junit4]   2> 1238935 DEBUG (simCloudManagerPool-1484-thread-17) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 
127.0.0.1:10013_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 
127.0.0.1:10007_solr]
   [junit4]   2> 1238935 DEBUG (simCloudManagerPool-1484-thread-17) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10010_solr, 
127.0.0.1:10013_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 
127.0.0.1:10007_solr]
   [junit4]   2> 1238935 DEBUG (simCloudManagerPool-1484-thread-17) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated 
with properties: {event=nodeAdded, waitFor=3, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}],
 enabled=true}
   [junit4]   2> 1238935 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 
13
   [junit4]   2> 1238936 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 13, lastZnodeVersion 13
   [junit4]   2> 1238949 DEBUG (ScheduledTrigger-1495-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 5
   [junit4]   2> 1238949 DEBUG (ScheduledTrigger-1495-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1238949 DEBUG (ScheduledTrigger-1495-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 5
   [junit4]   2> 1238969 DEBUG (ScheduledTrigger-1495-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 5
   [junit4]   2> 1238981 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[1F4E1C5F6521068F]) [    
] o.a.s.SolrTestCaseJ4 ###Ending testNodeAddedTrigger
   [junit4]   2> 1238981 DEBUG (ScheduledTrigger-1495-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1238982 DEBUG (ScheduledTrigger-1495-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 5
   [junit4]   2> 1239010 DEBUG (ScheduledTrigger-1495-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1239010 DEBUG (ScheduledTrigger-1495-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 5
   [junit4]   2> 1239010 DEBUG (ScheduledTrigger-1495-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 5
   [junit4]   2> 1239034 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[1F4E1C5F6521068F])
 [    ] o.a.s.SolrTestCaseJ4 ###Starting testNodeAddedTriggerRestoreState
   [junit4]   2> 1239034 DEBUG 
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[1F4E1C5F6521068F])
 [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 14
   [junit4]   2> 1239034 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 
14
   [junit4]   2> 1239034 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1239034 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1239034 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 14, lastZnodeVersion 14
   [junit4]   2> 1239034 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[1F4E1C5F6521068F])
 [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and 
clearing object cache...
   [junit4]   2> 1239035 WARN  (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Interrupted
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2>        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
   [junit4]   2>        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:166)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1239035 DEBUG 
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[1F4E1C5F6521068F])
 [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed 
explicitly
   [junit4]   2> 1239107 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 1239108 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 15
   [junit4]   2> 1239108 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 
127.0.0.1:10013_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 
127.0.0.1:10007_solr]
   [junit4]   2> 1239108 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 15, lastZnodeVersion -1
   [junit4]   2> 1239108 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 
15
   [junit4]   2> 1239128 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1239128 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 15, lastZnodeVersion 15
   [junit4]   2> 1239128 DEBUG (ScheduledTrigger-1497-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 5
   [junit4]   2> 1239165 DEBUG (simCloudManagerPool-1484-thread-18) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1239165 DEBUG (simCloudManagerPool-1484-thread-18) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
16
   [junit4]   2> 1239180 DEBUG (ScheduledTrigger-1497-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 5
   [junit4]   2> 1239201 DEBUG (simCloudManagerPool-1484-thread-18) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10010_solr, 
127.0.0.1:10013_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 
127.0.0.1:10007_solr]
   [junit4]   2> 1239201 DEBUG (simCloudManagerPool-1484-thread-18) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_restore_trigger 
instantiated with properties: {event=nodeAdded, waitFor=5, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}],
 enabled=true}
   [junit4]   2> 1239201 DEBUG (simCloudManagerPool-1484-thread-18) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 
127.0.0.1:10013_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 
127.0.0.1:10007_solr]
   [junit4]   2> 1239201 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 
16
   [junit4]   2> 1239201 INFO  (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 1239202 DEBUG (ScheduledTrigger-1497-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 6
   [junit4]   2> 1239202 DEBUG (ScheduledTrigger-1497-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
   [junit4]   2> 1239202 DEBUG (ScheduledTrigger-1497-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 6
   [junit4]   2> 1239202 DEBUG (ScheduledTrigger-1497-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10014_solr at time 
14046404093854365
   [junit4]   2> 1239217 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 16, lastZnodeVersion 16
   [junit4]   2> 1239225 DEBUG (ScheduledTrigger-1497-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 6
   [junit4]   2> 1239225 DEBUG (ScheduledTrigger-1497-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
   [junit4]   2> 1239225 DEBUG (ScheduledTrigger-1497-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 6
   [junit4]   2> 1239262 DEBUG (ScheduledTrigger-1497-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 6
   [junit4]   2> 1239262 DEBUG (ScheduledTrigger-1497-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
   [junit4]   2> 1239262 DEBUG (ScheduledTrigger-1497-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 6
   [junit4]   2> 1239263 DEBUG (simCloudManagerPool-1484-thread-19) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1239263 DEBUG (simCloudManagerPool-1484-thread-19) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
17
   [junit4]   2> 1239263 DEBUG (simCloudManagerPool-1484-thread-19) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10010_solr, 
127.0.0.1:10013_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 
127.0.0.1:10014_solr, 127.0.0.1:10007_solr]
   [junit4]   2> 1239263 DEBUG (simCloudManagerPool-1484-thread-19) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_restore_trigger 
instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}],
 enabled=true}
   [junit4]   2> 1239263 DEBUG (simCloudManagerPool-1484-thread-19) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 
127.0.0.1:10013_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 
127.0.0.1:10014_solr, 127.0.0.1:10007_solr]
   [junit4]   2> 1239274 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 
17
   [junit4]   2> 1239274 INFO  (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 1239275 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 17, lastZnodeVersion 17
   [junit4]   2> 1239275 DEBUG (ScheduledTrigger-1497-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
   [junit4]   2> 1239275 DEBUG (ScheduledTrigger-1497-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 6
   [junit4]   2> 1239275 DEBUG (ScheduledTrigger-1497-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_restore_trigger firing 
registered processor for nodes: [127.0.0.1:10014_solr] added at times 
[14046404093854365], now=14046407729598315
   [junit4]   2> 1239275 DEBUG (ScheduledTrigger-1497-thread-3) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"31e7214e62c29dTd2gueqt0mxaatvirnw4stvep2",
   [junit4]   2>   "source":"node_added_restore_trigger",
   [junit4]   2>   "eventTime":14046404093854365,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[14046404093854365],
   [junit4]   2>     "nodeNames":["127.0.0.1:10014_solr"]}}
   [junit4]   2> 1239315 DEBUG (AutoscalingActionExecutor-1498-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"31e7214e62c29dTd2gueqt0mxaatvirnw4stvep2",
   [junit4]   2>   "source":"node_added_restore_trigger",
   [junit4]   2>   "eventTime":14046404093854365,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[14046404093854365],
   [junit4]   2>     "_enqueue_time_":14046407736168815,
   [junit4]   2>     "nodeNames":["127.0.0.1:10014_solr"]}}
   [junit4]   2> 1239315 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[1F4E1C5F6521068F])
 [    ] o.a.s.SolrTestCaseJ4 ###Ending testNodeAddedTriggerRestoreState
   [junit4]   2> 1239400 DEBUG (AutoscalingActionExecutor-1498-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing took 4272 ms for event 
id=31e7214e62c29dTd2gueqt0mxaatvirnw4stvep2
   [junit4]   2> 1239412 DEBUG (ScheduledTrigger-1497-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
   [junit4]   2> 1239412 DEBUG (ScheduledTrigger-1497-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 6
   [junit4]   2> 1239412 DEBUG (ScheduledTrigger-1497-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 6
   [junit4]   2> 1239437 DEBUG (ScheduledTrigger-1497-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 6
   [junit4]   2> 1239457 DEBUG (ScheduledTrigger-1497-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 6
   [junit4]   2> 1239461 DEBUG (ScheduledTrigger-1497-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
   [junit4]   2> 1239461 DEBUG (ScheduledTrigger-1497-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 6
   [junit4]   2> 1239463 INFO  
(TEST-TestTriggerIntegration.testTriggerThrottling-seed#[1F4E1C5F6521068F]) [   
 ] o.a.s.SolrTestCaseJ4 ###Starting testTriggerThrottling
   [junit4]   2> 1239463 DEBUG 
(TEST-TestTriggerIntegration.testTriggerThrottling-seed#[1F4E1C5F6521068F]) [   
 ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 18
   [junit4]   2> 1239463 INFO  
(TEST-TestTriggerIntegration.testTriggerThrottling-seed#[1F4E1C5F6521068F]) [   
 ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and 
clearing object cache...
   [junit4]   2> 1239463 WARN  (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are 
closed, exiting.
   [junit4]   2> 1239463 DEBUG 
(TEST-TestTriggerIntegration.testTriggerThrottling-seed#[1F4E1C5F6521068F]) [   
 ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed 
explicitly
   [junit4]   2> 1239495 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 1239495 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 19
   [junit4]   2> 1239495 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 
127.0.0.1:10013_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 
127.0.0.1:10014_solr, 127.0.0.1:10007_solr]
   [junit4]   2> 1239495 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 19, lastZnodeVersion -1
   [junit4]   2> 1239495 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 
19
   [junit4]   2> 1239503 DEBUG (simCloudManagerPool-1484-thread-22) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1239503 DEBUG (simCloudManagerPool-1484-thread-22) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
20
   [junit4]   2> 1239503 DEBUG (simCloudManagerPool-1484-thread-22) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 
127.0.0.1:10013_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 
127.0.0.1:10014_solr, 127.0.0.1:10007_solr]
   [junit4]   2> 1239503 DEBUG (simCloudManagerPool-1484-thread-22) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10010_solr, 
127.0.0.1:10013_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 
127.0.0.1:10014_solr, 127.0.0.1:10007_solr]
   [junit4]   2> 1239503 DEBUG (simCloudManagerPool-1484-thread-22) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated 
with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$ThrottlingTesterAction}],
 enabled=true}
   [junit4]   2> 1239539 DEBUG (simCloudManagerPool-1484-thread-23) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1239539 DEBUG (simCloudManagerPool-1484-thread-23) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
21
   [junit4]   2> 1239539 DEBUG (simCloudManagerPool-1484-thread-23) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10010_solr, 
127.0.0.1:10013_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 
127.0.0.1:10014_solr, 127.0.0.1:10007_solr]
   [junit4]   2> 1239539 DEBUG (simCloudManagerPool-1484-thread-23) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 instantiated 
with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$ThrottlingTesterAction}],
 enabled=true}
   [junit4]   2> 1239539 DEBUG (simCloudManagerPool-1484-thread-23) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 
127.0.0.1:10013_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 
127.0.0.1:10014_solr, 127.0.0.1:10007_solr]
   [junit4]   2> 1239539 DEBUG (simCloudManagerPool-1484-thread-23) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10010_solr, 
127.0.0.1:10013_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 
127.0.0.1:10014_solr, 127.0.0.1:10007_solr]
   [junit4]   2> 1239539 DEBUG (simCloudManagerPool-1484-thread-23) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated 
with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$ThrottlingTesterAction}],
 enabled=true}
   [junit4]   2> 1239540 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1239540 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 21, lastZnodeVersion 19
   [junit4]   2> 1239540 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 
21
   [junit4]   2> 1239540 INFO  (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 1239540 DEBUG (ScheduledTrigger-1499-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 6
   [junit4]   2> 1239542 INFO  (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 1239543 DEBUG (ScheduledTrigger-1499-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1239543 DEBUG (ScheduledTrigger-1499-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 6
   [junit4]   2> 1239543 DEBUG (ScheduledTrigger-1499-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 1239543 DEBUG (ScheduledTrigger-1499-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 6
   [junit4]   2> 1239546 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 21, lastZnodeVersion 21
   [junit4]   2> 1239576 DEBUG (ScheduledTrigger-1499-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 7
   [junit4]   2> 1239576 DEBUG (ScheduledTrigger-1499-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1239576 DEBUG (ScheduledTrigger-1499-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 7
   [junit4]   2> 1239576 DEBUG (ScheduledTrigger-1499-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10015_solr at time 
14046422780481915
   [junit4]   2> 1239576 DEBUG (ScheduledTrigger-1499-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 firing 
registered processor for nodes: [127.0.0.1:10015_solr] added at times 
[14046422780481915], now=14046422782140465
   [junit4]   2> 1239576 DEBUG (ScheduledTrigger-1499-thread-2) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"31e725a832117bTd2gueqt0mxaatvirnw4stvep5",
   [junit4]   2>   "source":"node_added_trigger2",
   [junit4]   2>   "eventTime":14046422780481915,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[14046422780481915],
   [junit4]   2>     "nodeNames":["127.0.0.1:10015_solr"]}}
   [junit4]   2> 1239616 DEBUG (AutoscalingActionExecutor-1500-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"31e725a832117bTd2gueqt0mxaatvirnw4stvep5",
   [junit4]   2>   "source":"node_added_trigger2",
   [junit4]   2>   "eventTime":14046422780481915,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[14046422780481915],
   [junit4]   2>     "_enqueue_time_":14046422789327765,
   [junit4]   2>     "nodeNames":["127.0.0.1:10015_solr"]}}
   [junit4]   2> 1239616 INFO  (AutoscalingActionExecutor-1500-thread-1) [    ] 
o.a.s.c.a.s.TestTriggerIntegration action executed from node_added_trigger2
   [junit4]   2> 1239632 DEBUG (AutoscalingActionExecutor-1500-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing took 817 ms for event 
id=31e725a832117bTd2gueqt0mxaatvirnw4stvep5
   [junit4]   2> 1239669 DEBUG (ScheduledTrigger-1499-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 7
   [junit4]   2> 1239669 DEBUG (ScheduledTrigger-1499-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 1239669 DEBUG (ScheduledTrigger-1499-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 7
   [junit4]   2> 1239669 DEBUG (ScheduledTrigger-1499-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10015_solr at time 
14046427447293415
   [junit4]   2> 1239669 DEBUG (ScheduledTrigger-1499-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 firing 
registered processor for nodes: [127.0.0.1:10015_solr] added at times 
[14046427447293415], now=14046427449074715
   [junit4]   2> 1239669 DEBUG (ScheduledTrigger-1499-thread-3) [    ] 
o.a.s.c.a.ScheduledTriggers -------- Cooldown period - rejecting event: {
   [junit4]   2>   "id":"31e726be5bf5e7Td2gueqt0mxaatvirnw4stvep8",
   [junit4]   2>   "source":"node_added_trigger1",
   [junit4]   2>   "eventTime":14046427447293415,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[14046427447293415],
   [junit4]   2>     "nodeNames":["127.0.0.1:10015_solr"]}}
   [junit4]   2> 1239701 DEBUG (ScheduledTrigger-1499-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 7
   [junit4]   2> 1239738 DEBUG (ScheduledTrigger-1499-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 7
   [junit4]   2> 1239738 DEBUG (ScheduledTrigger-1499-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1239738 DEBUG (ScheduledTrigger-1499-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 7
   [junit4]   2> 1239760 DEBUG (ScheduledTrigger-1499-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 7
   [junit4]   2> 1239760 DEBUG (ScheduledTrigger-1499-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 1239760 DEBUG (ScheduledTrigger-1499-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 7
   [junit4]   2> 1239760 DEBUG (ScheduledTrigger-1499-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 firing 
registered processor for nodes: [127.0.0.1:10015_solr] added at times 
[14046427447293415], now=14046432002871215
   [junit4]   2> 1239760 DEBUG (ScheduledTrigger-1499-thread-2) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"31e726be5bf5e7Td2gueqt0mxaatvirnw4stvepa",
   [junit4]   2>   "source":"node_added_trigger1",
   [junit4]   2>   "eventTime":14046427447293415,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[14046427447293415],
   [junit4]   2>     "nodeNames":["127.0.0.1:10015_solr"]}}
   [junit4]   2> 1239777 DEBUG (AutoscalingActionExecutor-1500-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"31e726be5bf5e7Td2gueqt0mxaatvirnw4stvepa",
   [junit4]   2>   "source":"node_added_trigger1",
   [junit4]   2>   "eventTime":14046427447293415,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[14046427447293415],
   [junit4]   2>     "_enqueue_time_":14046432010901865,
   [junit4]   2>     "nodeNames":["127.0.0.1:10015_solr"]}}
   [junit4]   2> 1239777 INFO  (AutoscalingActionExecutor-1500-thread-1) [    ] 
o.a.s.c.ActionThrottle The last action attempt started 8070ms ago.
   [junit4]   2> 1239777 INFO  (AutoscalingActionExecutor-1500-thread-1) [    ] 
o.a.s.c.a.s.TestTriggerIntegration last action at 14046424793308965 time = 
14046432863020265
   [junit4]   2> 1239777 INFO  (AutoscalingActionExecutor-1500-thread-1) [    ] 
o.a.s.c.a.s.TestTriggerIntegration action executed from node_added_trigger1
   [junit4]   2> 1239795 DEBUG (AutoscalingActionExecutor-1500-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing took 884 ms for event 
id=31e726be5bf5e7Td2gueqt0mxaatvirnw4stvepa
   [junit4]   2> 1239815 DEBUG (ScheduledTrigger-1499-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 7
   [junit4]   2> 1239815 DEBUG (ScheduledTrigger-1499-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1239815 DEBUG (ScheduledTrigger-1499-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 7
   [junit4]   2> 1239815 DEBUG (ScheduledTrigger-1499-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 1239835 DEBUG (ScheduledTrigger-1499-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1239852 DEBUG (ScheduledTrigger-1499-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 7
   [junit4]   2> 1239835 DEBUG (ScheduledTrigger-1499-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 7
   [junit4]   2> 1239852 DEBUG (ScheduledTrigger-1499-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 7
   [junit4]   2> 1239824 DEBUG (simCloudManagerPool-1484-thread-29) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1239860 DEBUG (simCloudManagerPool-1484-thread-29) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
22
   [junit4]   2> 1239860 DEBUG (simCloudManagerPool-1484-thread-29) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10010_solr, 
127.0.0.1:10013_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 
127.0.0.1:10014_solr, 127.0.0.1:10007_solr, 127.0.0.1:10015_solr]
   [junit4]   2> 1239860 DEBUG (simCloudManagerPool-1484-thread-29) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 instantiated 
with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$ThrottlingTesterAction}],
 enabled=true}
   [junit4]   2> 1239860 DEBUG (simCloudManagerPool-1484-thread-29) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 
127.0.0.1:10013_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 
127.0.0.1:10014_solr, 127.0.0.1:10007_solr, 127.0.0.1:10015_solr]
   [junit4]   2> 1239860 DEBUG (simCloudManagerPool-1484-thread-29) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10010_solr, 
127.0.0.1:10013_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 
127.0.0.1:10014_solr, 127.0.0.1:10007_solr, 127.0.0.1:10015_solr]
   [junit4]   2> 1239860 DEBUG (simCloudManagerPool-1484-thread-29) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated 
with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$ThrottlingTesterAction}],
 enabled=true}
   [junit4]   2> 1239860 DEBUG (simCloudManagerPool-1484-thread-29) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0

[...truncated too long message...]

ulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread 
Processed trigger updates upto znodeVersion 44
   [junit4]   2> 1244558 DEBUG (AutoscalingActionExecutor-1512-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"31e7574867cbb9Td2gueqt0mxaatvirnw4stveq7",
   [junit4]   2>   "source":"search_rate_trigger",
   [junit4]   2>   "eventTime":14046635921755065,
   [junit4]   2>   "eventType":"SEARCHRATE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "node":{
   [junit4]   2>       "127.0.0.1:10018_solr":250.0,
   [junit4]   2>       "127.0.0.1:10014_solr":250.0},
   [junit4]   2>     "replica":[
   [junit4]   2>       "{\"core_node1\":{\n    
\"core\":\"collection1_shard1_replica_n1\",\n    
\"INDEX.sizeInBytes\":123450000,\n    \"rate\":250.0,\n    
\"node_name\":\"127.0.0.1:10018_solr\",\n    \"state\":\"active\",\n    
\"type\":\"NRT\",\n    \"QUERY./select.requestTimes:1minRate\":250.0,\n    
\"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}",
   [junit4]   2>       "{\"core_node2\":{\n    
\"core\":\"collection1_shard1_replica_n2\",\n    \"leader\":\"true\",\n    
\"INDEX.sizeInBytes\":123450000,\n    \"rate\":250.0,\n    
\"node_name\":\"127.0.0.1:10014_solr\",\n    \"state\":\"active\",\n    
\"type\":\"NRT\",\n    \"QUERY./select.requestTimes:1minRate\":250.0,\n    
\"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}"],
   [junit4]   2>     "collection":{"collection1":500.0},
   [junit4]   2>     "shard":{"collection1":{"shard1":500.0}},
   [junit4]   2>     "_enqueue_time_":14046657699804065}}
   [junit4]   2> 1244558 INFO  (AutoscalingActionExecutor-1512-thread-1) [    ] 
o.a.s.c.ActionThrottle The last action attempt started 35077ms ago.
   [junit4]   2> 1244558 WARN  (AutoscalingActionExecutor-1512-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Exception executing actions
   [junit4]   2> org.apache.lucene.store.AlreadyClosedException: 
ScheduledTrigger search_rate_trigger has been closed.
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers$ScheduledTrigger.dequeue(ScheduledTriggers.java:525)
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:337)
   [junit4]   2>        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2>        at 
java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1244558 DEBUG (AutoscalingActionExecutor-1512-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing took 11991 ms for event 
id=31e7574867cbb9Td2gueqt0mxaatvirnw4stveq7
   [junit4]   2> 1244566 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1244567 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 44, lastZnodeVersion 44
   [junit4]   2> 1244567 DEBUG (ScheduledTrigger-1513-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 8
   [junit4]   2> 1244591 DEBUG (simCloudManagerPool-1484-thread-64) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1244591 DEBUG (simCloudManagerPool-1484-thread-64) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
45
   [junit4]   2> 1244608 DEBUG (ScheduledTrigger-1513-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 8
   [junit4]   2> 1244608 DEBUG (simCloudManagerPool-1484-thread-64) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10018_solr, 
127.0.0.1:10010_solr, 127.0.0.1:10016_solr, 127.0.0.1:10017_solr, 
127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 127.0.0.1:10014_solr, 
127.0.0.1:10007_solr]
   [junit4]   2> 1244609 DEBUG (simCloudManagerPool-1484-thread-64) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10018_solr, 
127.0.0.1:10010_solr, 127.0.0.1:10016_solr, 127.0.0.1:10017_solr, 
127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 127.0.0.1:10014_solr, 
127.0.0.1:10007_solr]
   [junit4]   2> 1244609 DEBUG (simCloudManagerPool-1484-thread-64) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated 
with properties: {event=nodeAdded, waitFor=10, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}],
 enabled=true}
   [junit4]   2> 1244644 DEBUG (ScheduledTrigger-1513-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 8
   [junit4]   2> 1244644 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 
45
   [junit4]   2> 1244644 INFO  (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 1244644 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 45, lastZnodeVersion 45
   [junit4]   2> 1244644 INFO  
(TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[1F4E1C5F6521068F])
 [    ] o.a.s.SolrTestCaseJ4 ###Ending testEventFromRestoredState
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=TestTriggerIntegration -Dtests.method=testEventFromRestoredState 
-Dtests.seed=1F4E1C5F6521068F -Dtests.multiplier=2 -Dtests.slow=true 
-Dtests.locale=es-PA -Dtests.timezone=Etc/GMT+1 -Dtests.asserts=true 
-Dtests.file.encoding=US-ASCII
   [junit4] FAILURE 1.06s J1 | 
TestTriggerIntegration.testEventFromRestoredState <<<
   [junit4]    > Throwable #1: java.lang.AssertionError
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([1F4E1C5F6521068F:1F78A89F4C2CA2E5]:0)
   [junit4]    >        at 
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testEventFromRestoredState(TestTriggerIntegration.java:674)
   [junit4]    >        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1244645 DEBUG (ScheduledTrigger-1513-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1244645 DEBUG (ScheduledTrigger-1513-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 9
   [junit4]   2> 1244645 DEBUG (ScheduledTrigger-1513-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10020_solr at time 
14046676257320515
   [junit4]   2> 1244681 DEBUG 
(SUITE-TestTriggerIntegration-seed#[1F4E1C5F6521068F]-worker) [    ] 
o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 1244717 WARN  (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are 
closed, exiting.
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.sim.TestTriggerIntegration_1F4E1C5F6521068F-001
   [junit4]   2> NOTE: test params are: 
codec=FastCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST,
 chunkSize=8, maxDocsPerChunk=905, blockSize=75), 
termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST, 
chunkSize=8, blockSize=75)), sim=RandomSimilarity(queryNorm=false): {}, 
locale=es-PA, timezone=Etc/GMT+1
   [junit4]   2> NOTE: Linux 4.4.0-83-generic amd64/Oracle Corporation 
1.8.0_152 (64-bit)/cpus=4,threads=1,free=288371624,total=527433728
   [junit4]   2> NOTE: All tests run in this JVM: [MoveReplicaHDFSFailoverTest, 
TestAtomicUpdateErrorCases, TestLuceneMatchVersion, TestNestedDocsSort, 
TestClassicSimilarityFactory, SolrRequestParserTest, CdcrRequestHandlerTest, 
ShowFileRequestHandlerTest, DocValuesMultiTest, V2StandaloneTest, 
TestPolicyCloud, SolrGangliaReporterTest, DebugComponentTest, 
TestUninvertingReader, TestDeleteCollectionOnDownNodes, 
TestJavabinTupleStreamParser, SOLR749Test, RollingRestartTest, LargeFieldTest, 
TestDocumentBuilder, UUIDUpdateProcessorFallbackTest, SolrIndexMetricsTest, 
TestRemoteStreaming, DocumentAnalysisRequestHandlerTest, QueryEqualityTest, 
TestNodeLostTrigger, TestMinMaxOnMultiValuedField, TestFieldSortValues, 
SignatureUpdateProcessorFactoryTest, ReplaceNodeTest, 
CollectionsAPIDistributedZkTest, PreAnalyzedFieldTest, DeleteShardTest, 
ChaosMonkeySafeLeaderWithPullReplicasTest, TestSchemalessBufferedUpdates, 
AddReplicaTest, ZkFailoverTest, PeerSyncReplicationTest, TestRawTransformer, 
TestTriggerIntegration]
   [junit4] Completed [134/763 (1!)] on J1 in 7.63s, 11 tests, 3 failures <<< 
FAILURES!

[...truncated 40458 lines...]
BUILD FAILED
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/build.xml:835: The 
following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/build.xml:101: The 
following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build.xml:142:
 The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build.xml:499:
 The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/common-build.xml:2630:
 Failed to load 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/dev-tools/doap/lucene.rdf

Total time: 276 minutes 50 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to