[JENKINS] Lucene-Solr-7.x-Linux (64bit/jdk-11) - Build # 3296 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/3296/
Java: 64bit/jdk-11 -XX:+UseCompressedOops -XX:+UseSerialGC

2 tests failed.
FAILED:  
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testNodeLostTriggerRestoreState

Error Message:
The trigger did not fire at all

Stack Trace:
java.lang.AssertionError: The trigger did not fire at all
        at 
__randomizedtesting.SeedInfo.seed([4444F3BA3046D308:6FBB26E1AA3EC6D8]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.assertTrue(Assert.java:41)
        at 
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testNodeLostTriggerRestoreState(TestSimTriggerIntegration.java:332)
        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:834)


FAILED:  
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testNodeMarkersRegistration

Error Message:
Path /autoscaling/nodeAdded/127.0.0.1:10003_solr should have been deleted

Stack Trace:
java.lang.AssertionError: Path /autoscaling/nodeAdded/127.0.0.1:10003_solr 
should have been deleted
        at 
__randomizedtesting.SeedInfo.seed([4444F3BA3046D308:5CFE7BB63E731EE7]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.assertTrue(Assert.java:41)
        at org.junit.Assert.assertFalse(Assert.java:64)
        at 
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testNodeMarkersRegistration(TestSimTriggerIntegration.java:892)
        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:834)




Build Log:
[...truncated 12662 lines...]
   [junit4] Suite: 
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration
   [junit4]   2> Creating dataDir: 
/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.sim.TestSimTriggerIntegration_4444F3BA3046D308-001/init-core-data-001
   [junit4]   2> 47464 INFO  
(SUITE-TestSimTriggerIntegration-seed#[4444F3BA3046D308]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) 
w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 47465 INFO  
(SUITE-TestSimTriggerIntegration-seed#[4444F3BA3046D308]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: 
@org.apache.solr.util.RandomizeSSL(reason="", ssl=0.0/0.0, value=0.0/0.0, 
clientAuth=0.0/0.0)
   [junit4]   2> 47465 INFO  
(SUITE-TestSimTriggerIntegration-seed#[4444F3BA3046D308]-worker) [    ] 
o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: 
test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4] IGNOR/A 0.00s J2 | TestSimTriggerIntegration.testSearchRate
   [junit4]    > Assumption #1: 'awaitsfix' test group is disabled 
(@AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";))
   [junit4] IGNOR/A 0.00s J2 | 
TestSimTriggerIntegration.testNodeMarkersRegistration
   [junit4]    > Assumption #1: 'badapple' test group is disabled 
(@BadApple(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";))
   [junit4] IGNOR/A 0.00s J2 | TestSimTriggerIntegration.testCooldown
   [junit4]    > Assumption #1: 'badapple' test group is disabled 
(@BadApple(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";))
   [junit4] IGNOR/A 0.00s J2 | TestSimTriggerIntegration.testEventQueue
   [junit4]    > Assumption #1: 'awaitsfix' test group is disabled 
(@AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";))
   [junit4] IGNOR/A 0.00s J2 | 
TestSimTriggerIntegration.testNodeAddedTriggerRestoreState
   [junit4]    > Assumption #1: 'badapple' test group is disabled 
(@BadApple(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";))
   [junit4]   2> 47471 INFO  
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[4444F3BA3046D308])
 [    ] o.a.s.SolrTestCaseJ4 ###Starting testNodeLostTriggerRestoreState
   [junit4]   2> 47552 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[4444F3BA3046D308])
 [    ] o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader: 
127.0.0.1:10000_solr
   [junit4]   2> 47556 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[4444F3BA3046D308])
 [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, 
currentVersion=0
   [junit4]   2> 47558 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[4444F3BA3046D308])
 [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 0
   [junit4]   2> 47558 INFO  
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[4444F3BA3046D308])
 [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics 
history in memory.
   [junit4]   2> 47563 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and 
.scheduled_maintenance triggers
   [junit4]   2> 47565 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 1
   [junit4]   2> 47625 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 0
   [junit4]   2> 47625 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 47645 DEBUG (simCloudManagerPool-321-thread-2) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 47652 DEBUG (simCloudManagerPool-321-thread-2) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
2
   [junit4]   2> 47657 DEBUG (simCloudManagerPool-321-thread-3) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 47657 DEBUG (simCloudManagerPool-321-thread-3) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
3
   [junit4]   2> 47659 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[4444F3BA3046D308])
 [    ] o.a.s.c.a.s.SimClusterStateProvider  -- created marker: 
/autoscaling/nodeLost/127.0.0.1:10002_solr
   [junit4]   2> 47659 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial 
livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
   [junit4]   2> 47660 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeLostTrigger Adding lost node from marker path: 
127.0.0.1:10002_solr
   [junit4]   2> 47679 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 47679 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 1
   [junit4]   2> 47679 DEBUG (ScheduledTrigger-322-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 47679 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 47717 DEBUG (ScheduledTrigger-322-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 47717 INFO  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.s.TestSimTriggerIntegration TestTriggerAction init
   [junit4]   2> 47717 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeLostTrigger NodeLostTrigger node_lost_restore_trigger - Initial 
livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
   [junit4]   2> 47717 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 47717 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 47717 DEBUG (ScheduledTrigger-322-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 47718 DEBUG (simCloudManagerPool-321-thread-4) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 47718 DEBUG (simCloudManagerPool-321-thread-4) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
4
   [junit4]   2> 47719 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 47720 INFO  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.s.TestSimTriggerIntegration TestTriggerAction init
   [junit4]   2> 47720 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeLostTrigger NodeLostTrigger node_lost_restore_trigger - Initial 
livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
   [junit4]   2> 47720 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 47720 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
   [junit4]   2> 47720 DEBUG (ScheduledTrigger-322-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 47737 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 47740 DEBUG (ScheduledTrigger-322-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 47757 DEBUG (ScheduledTrigger-322-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 47760 DEBUG (ScheduledTrigger-322-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 47778 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 47781 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 47798 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 47801 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 47818 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 47821 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 47839 DEBUG (ScheduledTrigger-322-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 47842 DEBUG (ScheduledTrigger-322-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 47859 DEBUG (ScheduledTrigger-322-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 47862 DEBUG (ScheduledTrigger-322-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 47879 DEBUG (ScheduledTrigger-322-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 47882 DEBUG (ScheduledTrigger-322-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 47900 DEBUG (ScheduledTrigger-322-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 47903 DEBUG (ScheduledTrigger-322-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 47927 DEBUG (ScheduledTrigger-322-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 47927 DEBUG (ScheduledTrigger-322-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 47950 DEBUG (ScheduledTrigger-322-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 47951 DEBUG (ScheduledTrigger-322-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 47971 DEBUG (ScheduledTrigger-322-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 47971 DEBUG (ScheduledTrigger-322-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 47991 DEBUG (ScheduledTrigger-322-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 47991 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48011 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48012 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48032 DEBUG (ScheduledTrigger-322-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48032 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48052 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48052 DEBUG (ScheduledTrigger-322-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48072 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48072 DEBUG (ScheduledTrigger-322-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48093 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48093 DEBUG (ScheduledTrigger-322-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48114 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48114 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48134 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48134 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48154 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48154 DEBUG (ScheduledTrigger-322-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48175 DEBUG (ScheduledTrigger-322-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48175 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48195 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48195 DEBUG (ScheduledTrigger-322-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48215 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48215 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48236 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48236 DEBUG (ScheduledTrigger-322-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48256 DEBUG (ScheduledTrigger-322-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48256 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48276 DEBUG (ScheduledTrigger-322-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48276 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48296 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48297 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48317 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48317 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48337 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48337 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48357 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48358 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48378 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48378 DEBUG (ScheduledTrigger-322-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48398 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48399 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48419 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48419 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48439 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48439 DEBUG (ScheduledTrigger-322-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48460 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48460 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48480 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48480 DEBUG (ScheduledTrigger-322-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48500 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48500 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48520 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48521 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48541 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48541 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48561 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48561 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48583 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48583 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48603 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48603 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48624 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48624 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48644 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48644 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48664 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48664 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48686 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48686 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48707 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48707 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48727 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48728 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48748 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48748 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48771 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48771 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48791 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48792 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48812 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48812 DEBUG (ScheduledTrigger-322-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48832 DEBUG (ScheduledTrigger-322-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48832 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48852 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48852 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48873 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48873 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48893 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48893 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48913 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48913 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48933 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48933 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48954 DEBUG (ScheduledTrigger-322-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48954 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48974 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48974 DEBUG (ScheduledTrigger-322-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48994 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 48995 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49015 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49015 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49036 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49036 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49056 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49056 DEBUG (ScheduledTrigger-322-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49076 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49076 DEBUG (ScheduledTrigger-322-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49096 DEBUG (ScheduledTrigger-322-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49096 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49117 DEBUG (ScheduledTrigger-322-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49117 DEBUG (ScheduledTrigger-322-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49137 DEBUG (ScheduledTrigger-322-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49137 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49157 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49158 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49178 DEBUG (ScheduledTrigger-322-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49178 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49198 DEBUG (ScheduledTrigger-322-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49198 DEBUG (ScheduledTrigger-322-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49218 DEBUG (ScheduledTrigger-322-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49218 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49239 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49239 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49259 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49259 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49279 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49279 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49299 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49299 DEBUG (ScheduledTrigger-322-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49320 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49320 DEBUG (ScheduledTrigger-322-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49340 DEBUG (ScheduledTrigger-322-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49340 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49360 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49360 DEBUG (ScheduledTrigger-322-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49381 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49381 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49401 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49401 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49421 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49422 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49442 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49442 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49464 DEBUG (ScheduledTrigger-322-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49464 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49486 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49486 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49506 DEBUG (ScheduledTrigger-322-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49506 DEBUG (ScheduledTrigger-322-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 49520 INFO  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are 
closed, exiting.
   [junit4]   2> 49521 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[4444F3BA3046D308])
 [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool 
executor now
   [junit4]   2> 49521 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[4444F3BA3046D308])
 [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 49521 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[4444F3BA3046D308])
 [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 49521 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[4444F3BA3046D308])
 [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread 
pool executor
   [junit4]   2> 49521 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[4444F3BA3046D308])
 [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 49521 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[4444F3BA3046D308])
 [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed 
explicitly
   [junit4]   2> 49522 INFO  
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[4444F3BA3046D308])
 [    ] o.a.s.SolrTestCaseJ4 ###Ending testNodeLostTriggerRestoreState
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=TestSimTriggerIntegration 
-Dtests.method=testNodeLostTriggerRestoreState -Dtests.seed=4444F3BA3046D308 
-Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en-SE 
-Dtests.timezone=Asia/Seoul -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 2.06s J2 | 
TestSimTriggerIntegration.testNodeLostTriggerRestoreState <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: The trigger did not 
fire at all
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([4444F3BA3046D308:6FBB26E1AA3EC6D8]:0)
   [junit4]    >        at 
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testNodeLostTriggerRestoreState(TestSimTriggerIntegration.java:332)
   [junit4]    >        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]    >        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]    >        at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]    >        at 
java.base/java.lang.reflect.Method.invoke(Method.java:566)
   [junit4]    >        at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4] IGNOR/A 0.00s J2 | TestSimTriggerIntegration.testTriggerThrottling
   [junit4]    > Assumption #1: 'badapple' test group is disabled 
(@BadApple(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";))
   [junit4] IGNOR/A 0.00s J2 | TestSimTriggerIntegration.testListeners
   [junit4]    > Assumption #1: 'badapple' test group is disabled 
(@BadApple(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";))
   [junit4] IGNOR/A 0.00s J2 | TestSimTriggerIntegration.testNodeLostTrigger
   [junit4]    > Assumption #1: 'awaitsfix' test group is disabled 
(@AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";))
   [junit4] IGNOR/A 0.00s J2 | 
TestSimTriggerIntegration.testEventFromRestoredState
   [junit4]    > Assumption #1: 'badapple' test group is disabled 
(@BadApple(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";))
   [junit4] IGNOR/A 0.00s J2 | TestSimTriggerIntegration.testNodeAddedTrigger
   [junit4]    > Assumption #1: 'badapple' test group is disabled 
(@BadApple(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";))
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.sim.TestSimTriggerIntegration_4444F3BA3046D308-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, 
docValues:{}, maxPointsInLeafNode=494, maxMBSortInHeap=5.063057871809752, 
sim=RandomSimilarity(queryNorm=true): {}, locale=en-SE, timezone=Asia/Seoul
   [junit4]   2> NOTE: Linux 4.15.0-42-generic amd64/Oracle Corporation 11 
(64-bit)/cpus=8,threads=1,free=270978104,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [TaggingAttributeTest, 
TermsComponentTest, TestStressUserVersions, DistributedTermsComponentTest, 
TestInitParams, SolrCmdDistributorTest, QueryEqualityTest, 
TestSchemaNameResource, ZkNodePropsTest, TestHdfsCloudBackupRestore, 
TestSimTriggerIntegration]
   [junit4] Completed [19/836 (1!)] on J2 in 2.27s, 11 tests, 1 failure, 10 
skipped <<< FAILURES!

[...truncated 45315 lines...]
[repro] Jenkins log URL: 
https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/3296/consoleText

[repro] Revision: 9aa15a2accd47214dc2d76a035fa31450a079f62

[repro] Ant options: "-Dargs=-XX:+UseCompressedOops -XX:+UseSerialGC"
[repro] Repro line:  ant test  -Dtestcase=TestSimTriggerIntegration 
-Dtests.method=testNodeLostTriggerRestoreState -Dtests.seed=4444F3BA3046D308 
-Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en-SE 
-Dtests.timezone=Asia/Seoul -Dtests.asserts=true -Dtests.file.encoding=UTF-8

[repro] ant clean

[...truncated 6 lines...]
[repro] Test suites by module:
[repro]    solr/core
[repro]       TestSimTriggerIntegration
[repro] ant compile-test

[...truncated 3610 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=5 
-Dtests.class="*.TestSimTriggerIntegration" -Dtests.showOutput=onerror 
"-Dargs=-XX:+UseCompressedOops -XX:+UseSerialGC" -Dtests.seed=4444F3BA3046D308 
-Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en-SE 
-Dtests.timezone=Asia/Seoul -Dtests.asserts=true -Dtests.file.encoding=UTF-8

[...truncated 32 lines...]
   [junit4] Suite: 
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration
   [junit4]   2> Creating dataDir: 
/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.sim.TestSimTriggerIntegration_4444F3BA3046D308-001/init-core-data-001
   [junit4]   2> 2197 INFO  
(SUITE-TestSimTriggerIntegration-seed#[4444F3BA3046D308]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) 
w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 2253 INFO  
(SUITE-TestSimTriggerIntegration-seed#[4444F3BA3046D308]-worker) [    ] 
o.e.j.u.log Logging initialized @2274ms to org.eclipse.jetty.util.log.Slf4jLog
   [junit4]   2> 2265 INFO  
(SUITE-TestSimTriggerIntegration-seed#[4444F3BA3046D308]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: 
@org.apache.solr.util.RandomizeSSL(reason="", ssl=0.0/0.0, value=0.0/0.0, 
clientAuth=0.0/0.0)
   [junit4]   2> 2271 INFO  
(SUITE-TestSimTriggerIntegration-seed#[4444F3BA3046D308]-worker) [    ] 
o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: 
test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4] IGNOR/A 0.00s J2 | TestSimTriggerIntegration.testSearchRate
   [junit4]    > Assumption #1: 'awaitsfix' test group is disabled 
(@AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";))
   [junit4]   2> 2303 INFO  
(TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[4444F3BA3046D308])
 [    ] o.a.s.SolrTestCaseJ4 ###Starting testNodeMarkersRegistration
   [junit4]   2> 2403 INFO  
(TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[4444F3BA3046D308])
 [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not 
find system property or JNDI)
   [junit4]   2> 2502 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[4444F3BA3046D308])
 [    ] o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader: 
127.0.0.1:10000_solr
   [junit4]   2> 2582 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[4444F3BA3046D308])
 [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, 
currentVersion=0
   [junit4]   2> 2569 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] 
o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and 
.scheduled_maintenance triggers
   [junit4]   2> 2612 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
1
   [junit4]   2> 2595 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[4444F3BA3046D308])
 [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 0
   [junit4]   2> 2612 INFO  
(TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[4444F3BA3046D308])
 [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics 
history in memory.
   [junit4]   2> 2690 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] 
o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 0
   [junit4]   2> 2692 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] 
o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 2733 DEBUG (simCloudManagerPool-2-thread-2) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 2735 DEBUG (simCloudManagerPool-2-thread-2) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
2
   [junit4]   2> 2746 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] 
o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial 
livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
   [junit4]   2> 2748 INFO  
(TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[4444F3BA3046D308])
 [    ] o.a.s.c.a.s.TestSimTriggerIntegration ====== KILL OVERSEER 1
   [junit4]   2> 2749 INFO  
(TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[4444F3BA3046D308])
 [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and 
clearing object cache...
   [junit4]   2> 2765 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] 
o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 2765 DEBUG (ScheduledTrigger-3-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3 and last live nodes: 2
   [junit4]   2> 2766 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[4444F3BA3046D308])
 [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool 
executor now
   [junit4]   2> 2766 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[4444F3BA3046D308])
 [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 2766 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[4444F3BA3046D308])
 [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 2767 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[4444F3BA3046D308])
 [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread 
pool executor
   [junit4]   2> 2767 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[4444F3BA3046D308])
 [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 2767 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[4444F3BA3046D308])
 [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed 
explicitly
   [junit4]   2> 2767 INFO  
(TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[4444F3BA3046D308])
 [    ] o.a.s.c.a.s.SimCloudManager   = killing node 127.0.0.1:10000_solr
   [junit4]   2> 2768 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[4444F3BA3046D308])
 [    ] o.a.s.c.a.s.SimClusterStateProvider  -- created marker: 
/autoscaling/nodeLost/127.0.0.1:10000_solr
   [junit4]   2> 2769 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[4444F3BA3046D308])
 [    ] o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader: 
127.0.0.1:10001_solr
   [junit4]   2> 2783 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
2
   [junit4]   2> 2784 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] 
o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 0
   [junit4]   2> 2784 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] 
o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 2786 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] 
o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial 
livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10002_solr]
   [junit4]   2> 2786 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] 
o.a.s.c.a.NodeLostTrigger Adding lost node from marker path: 
127.0.0.1:10000_solr
   [junit4]   2> 2801 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] 
o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 2801 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] 
o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
   [junit4]   2> 2813 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 3
   [junit4]   2> 2814 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10000_solr
   [junit4]   2> 2834 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 2855 DEBUG (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 2875 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 2882 INFO  
(TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[4444F3BA3046D308])
 [    ] o.a.s.c.a.s.TestSimTriggerIntegration ====== ADD TRIGGERS
   [junit4]   2> 2885 INFO  (simCloudManagerPool-7-thread-1) [    ] 
o.a.s.c.a.s.TestSimTriggerIntegration TestEventMarkerAction init
   [junit4]   2> 2887 DEBUG (simCloudManagerPool-7-thread-1) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 2888 DEBUG (simCloudManagerPool-7-thread-1) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
3
   [junit4]   2> 2889 INFO  (simCloudManagerPool-7-thread-1) [    ] 
o.a.s.c.a.s.TestSimTriggerIntegration TestEventMarkerAction init
   [junit4]   2> 2889 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] 
o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 2891 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger - Initial 
livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10002_solr]
   [junit4]   2> 2892 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated 
with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration$TestEventMarkerAction}],
 enabled=true}
   [junit4]   2> 2892 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] 
o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 2893 DEBUG (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 2893 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] 
o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 2893 DEBUG (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 2896 DEBUG (ScheduledTrigger-8-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 2898 INFO  (simCloudManagerPool-7-thread-2) [    ] 
o.a.s.c.a.s.TestSimTriggerIntegration TestEventMarkerAction init
   [junit4]   2> 2899 DEBUG (simCloudManagerPool-7-thread-2) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 2900 DEBUG (simCloudManagerPool-7-thread-2) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
4
   [junit4]   2> 2901 INFO  (simCloudManagerPool-7-thread-2) [    ] 
o.a.s.c.a.s.TestSimTriggerIntegration TestEventMarkerAction init
   [junit4]   2> 2901 INFO  (simCloudManagerPool-7-thread-2) [    ] 
o.a.s.c.a.s.TestSimTriggerIntegration TestEventMarkerAction init
   [junit4]   2> 2901 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] 
o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 2901 INFO  
(TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[4444F3BA3046D308])
 [    ] o.a.s.c.a.s.TestSimTriggerIntegration ====== ADD NODE 1
   [junit4]   2> 2902 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] 
o.a.s.c.a.NodeLostTrigger NodeLostTrigger node_lost_trigger - Initial 
livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10002_solr]
   [junit4]   2> 2902 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] 
o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 2902 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] 
o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
   [junit4]   2> 2902 DEBUG (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 2903 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeMarkersRegistration-seed#[4444F3BA3046D308])
 [    ] o.a.s.c.a.s.SimClusterStateProvider -- creating marker: 
/autoscaling/nodeAdded/127.0.0.1:10003_solr
   [junit4]   2> 2914 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 2914 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 2914 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10003_solr at time 
982310672751288
   [junit4]   2> 2920 DEBUG (ScheduledTrigger-8-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3 and last live nodes: 2
   [junit4]   2> 2923 DEBUG (ScheduledTrigger-8-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 3 and last live nodes: 2
   [junit4]   2> 2935 DEBUG (ScheduledTrigger-8-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 2935 DEBUG (ScheduledTrigger-8-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 2935 DEBUG (ScheduledTrigger-8-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing 
registered processor for nodes: [127.0.0.1:10003_solr] added at times 
[982310672751288], now=982311713097488
   [junit4]   2> 2937 DEBUG (ScheduledTrigger-8-thread-3) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"37d6806b55ab8T9hcv6ggjwa2gt0u7yah8t3xhc",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":982310672751288,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[982310672751288],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "nodeNames":["127.0.0.1:10003_solr"]}}
   [junit4]   2> 2938 DEBUG (ScheduledTrigger-8-thread-3) [    ] 
o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, 
node_lost_trigger, node_added_trigger]
   [junit4]   2> 2942 DEBUG (simCloudManagerPool-7-thread-3) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- simCreateCollection .system, 
currentVersion=1
   [junit4]   2> 2947 DEBUG (simCloudManagerPool-7-thread-3) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, 
currentVersion=1
   [junit4]   2> 2947 DEBUG (simCloudManagerPool-7-thread-3) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 1
   [junit4]   2> 2967 DEBUG (simCloudManagerPool-7-thread-3) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, 
currentVersion=2
   [junit4]   2> 2970 DEBUG (simCloudManagerPool-7-thread-3) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 2
   [junit4]   2> 3034 DEBUG (simCloudManagerPool-7-thread-4) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, 
currentVersion=3
   [junit4]   2> 3035 DEBUG (simCloudManagerPool-7-thread-4) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 3
   [junit4]   2> 3048 DEBUG (simCloudManagerPool-7-thread-5) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, 
currentVersion=4
   [junit4]   2> 3049 DEBUG (simCloudManagerPool-7-thread-5) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 4
   [junit4]   2> 3060 DEBUG (simCloudManagerPool-7-thread-6) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, 
currentVersion=5
   [junit4]   2> 3060 DEBUG (simCloudManagerPool-7-thread-6) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 5
   [junit4]   2> 3061 DEBUG (simCloudManagerPool-7-thread-3) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- finished createCollection .system, 
currentVersion=6
   [junit4]   2> 3062 INFO  (simCloudManagerPool-7-thread-3) [    ] 
o.a.s.c.CloudTestUtils -- shards without leaders: [shard1]
   [junit4]   2> 3064 INFO  (simCloudManagerPool-7-thread-3) [    ] 
o.a.s.c.CloudTestUtils -- shards without leaders: [shard1]
   [junit4]   2> 3063 DEBUG (simCloudManagerPool-7-thread-7) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for .system / shard1 
(currentVersion=6): {"core_node1":{
   [junit4]   2>     "core":".system_shard1_replica_n1",
   [junit4]   2>     "shard":"shard1",
   [junit4]   2>     "collection":".system",
   [junit4]   2>     "node_name":"127.0.0.1:10003_solr",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":10240,
   [junit4]   2>     "state":"active",
   [junit4]   2>     "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>     "SEARCHER.searcher.numDocs":0}}
   [junit4]   2> 3067 DEBUG (simCloudManagerPool-7-thread-3) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, 
currentVersion=6
   [junit4]   2> 3067 DEBUG (simCloudManagerPool-7-thread-3) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 6
   [junit4]   2> 3072 DEBUG (ScheduledTrigger-8-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Removing new node from tracking: 127.0.0.1:10003_solr
   [junit4]   2> 3072 DEBUG (AutoscalingActionExecutor-9-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"37d6806b55ab8T9hcv6ggjwa2gt0u7yah8t3xhc",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":982310672751288,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[982310672751288],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "_enqueue_time_":982311879284438,
   [junit4]   2>     "nodeNames":["127.0.0.1:10003_solr"]}}
   [junit4]   2> 3075 DEBUG (AutoscalingActionExecutor-9-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
   [junit4]   2> 3075 DEBUG (AutoscalingActionExecutor-9-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: node_lost_trigger after 100ms
   [junit4]   2> 3076 DEBUG (AutoscalingActionExecutor-9-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger after 100ms
   [junit4]   2> 3076 DEBUG (AutoscalingActionExecutor-9-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing took 208 ms for event 
id=37d6806b55ab8T9hcv6ggjwa2gt0u7yah8t3xhc
   [junit4]   2> 3176 DEBUG (ScheduledTrigger-8-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3176 DEBUG (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3176 DEBUG (ScheduledTrigger-8-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 3176 DEBUG (ScheduledTrigger-8-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 3197 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3197 DEBUG (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3197 DEBUG (ScheduledTrigger-8-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 3197 DEBUG (ScheduledTrigger-8-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 3217 DEBUG (ScheduledTrigger-8-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3217 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3217 DEBUG (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 3218 DEBUG (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 3238 DEBUG (ScheduledTrigger-8-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 3238 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3238 DEBUG (ScheduledTrigger-8-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 3238 DEBUG (ScheduledTrigger-8-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3258 DEBUG (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 3258 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3259 DEBUG (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 3259 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3279 DEBUG (ScheduledTrigger-8-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 3279 DEBUG (ScheduledTrigger-8-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3279 DEBUG (ScheduledTrigger-8-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 3279 DEBUG (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3300 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 3300 DEBUG (ScheduledTrigger-8-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3300 DEBUG (ScheduledTrigger-8-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3300 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 3320 DEBUG (ScheduledTrigger-8-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3320 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3320 DEBUG (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 3320 DEBUG (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 3341 DEBUG (ScheduledTrigger-8-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3341 DEBUG (ScheduledTrigger-8-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3341 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 3341 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 3361 DEBUG (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 3361 DEBUG (ScheduledTrigger-8-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3361 DEBUG (ScheduledTrigger-8-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3361 DEBUG (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 3382 DEBUG (ScheduledTrigger-8-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 3382 DEBUG (ScheduledTrigger-8-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3382 DEBUG (ScheduledTrigger-8-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 3382 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3402 DEBUG (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3402 DEBUG (ScheduledTrigger-8-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 3402 DEBUG (ScheduledTrigger-8-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3402 DEBUG (ScheduledTrigger-8-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 3423 DEBUG (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3423 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3423 DEBUG (ScheduledTrigger-8-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 3423 DEBUG (ScheduledTrigger-8-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 3443 DEBUG (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3443 DEBUG (ScheduledTrigger-8-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3443 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 3444 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 3464 DEBUG (ScheduledTrigger-8-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 3464 DEBUG (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3464 DEBUG (ScheduledTrigger-8-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3464 DEBUG (ScheduledTrigger-8-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 3484 DEBUG (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3484 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3484 DEBUG (ScheduledTrigger-8-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 3485 DEBUG (ScheduledTrigger-8-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 3505 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 3505 DEBUG (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3505 DEBUG (ScheduledTrigger-8-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3505 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 3525 DEBUG (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3525 DEBUG (ScheduledTrigger-8-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3525 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 3525 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 3546 DEBUG (ScheduledTrigger-8-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 3546 DEBUG (ScheduledTrigger-8-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3546 DEBUG (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3546 DEBUG (ScheduledTrigger-8-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 3566 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3566 DEBUG (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 3566 DEBUG (ScheduledTrigger-8-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3566 DEBUG (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 3587 DEBUG (ScheduledTrigger-8-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 3587 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3587 DEBUG (ScheduledTrigger-8-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3587 DEBUG (ScheduledTrigger-8-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 3607 DEBUG (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 3607 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3607 DEBUG (ScheduledTrigger-8-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3607 DEBUG (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 3628 DEBUG (ScheduledTrigger-8-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 3628 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3628 DEBUG (ScheduledTrigger-8-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 3628 DEBUG (ScheduledTrigger-8-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3648 DEBUG (ScheduledTrigger-8-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 3648 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3 and last live nodes: 3
   [junit4]   2> 3648 DEBUG (ScheduledTrigger-8-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 3648 DEBUG (ScheduledTrigger-8-thread-2) [    ] o.a.s.c.

[...truncated too long message...]

0.1:10027_solr] added at times [982516729615594], now=982519778187844
   [junit4]   2> 7386 DEBUG (ScheduledTrigger-54-thread-2) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"37d9800a794eaT9hcv6ggjwa2gt0u7yah8t3xij",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":982516729615594,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[982516729615594],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "nodeNames":["127.0.0.1:10027_solr"]}}
   [junit4]   2> 7387 DEBUG (ScheduledTrigger-54-thread-2) [    ] 
o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, 
node_added_trigger]
   [junit4]   2> 7387 DEBUG (simCloudManagerPool-53-thread-4) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- simCreateCollection .system, 
currentVersion=1
   [junit4]   2> 7388 DEBUG (simCloudManagerPool-53-thread-4) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, 
currentVersion=1
   [junit4]   2> 7388 DEBUG (simCloudManagerPool-53-thread-4) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 1
   [junit4]   2> 7398 DEBUG (simCloudManagerPool-53-thread-4) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, 
currentVersion=2
   [junit4]   2> 7398 DEBUG (simCloudManagerPool-53-thread-4) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 2
   [junit4]   2> 7410 DEBUG (simCloudManagerPool-53-thread-5) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, 
currentVersion=3
   [junit4]   2> 7410 DEBUG (simCloudManagerPool-53-thread-5) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 3
   [junit4]   2> 7420 DEBUG (simCloudManagerPool-53-thread-6) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, 
currentVersion=4
   [junit4]   2> 7421 DEBUG (simCloudManagerPool-53-thread-6) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 4
   [junit4]   2> 7431 DEBUG (simCloudManagerPool-53-thread-7) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, 
currentVersion=5
   [junit4]   2> 7432 DEBUG (simCloudManagerPool-53-thread-7) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 5
   [junit4]   2> 7432 DEBUG (simCloudManagerPool-53-thread-4) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- finished createCollection .system, 
currentVersion=6
   [junit4]   2> 7432 INFO  (simCloudManagerPool-53-thread-4) [    ] 
o.a.s.c.CloudTestUtils -- shards without leaders: [shard1]
   [junit4]   2> 7432 DEBUG (simCloudManagerPool-53-thread-10) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for .system / shard1 
(currentVersion=6): {"core_node1":{
   [junit4]   2>     "core":".system_shard1_replica_n1",
   [junit4]   2>     "shard":"shard1",
   [junit4]   2>     "collection":".system",
   [junit4]   2>     "node_name":"127.0.0.1:10027_solr",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":10240,
   [junit4]   2>     "state":"active",
   [junit4]   2>     "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>     "SEARCHER.searcher.numDocs":0}}
   [junit4]   2> 7434 DEBUG (simCloudManagerPool-53-thread-4) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, 
currentVersion=6
   [junit4]   2> 7435 DEBUG (simCloudManagerPool-53-thread-4) [    ] 
o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 6
   [junit4]   2> 7435 DEBUG (ScheduledTrigger-54-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Removing new node from tracking: 127.0.0.1:10027_solr
   [junit4]   2> 7435 DEBUG (AutoscalingActionExecutor-55-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"37d9800a794eaT9hcv6ggjwa2gt0u7yah8t3xij",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":982516729615594,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[982516729615594],
   [junit4]   2>     "preferredOperation":"movereplica",
   [junit4]   2>     "_enqueue_time_":982519805717694,
   [junit4]   2>     "nodeNames":["127.0.0.1:10027_solr"]}}
   [junit4]   2> 7436 DEBUG (AutoscalingActionExecutor-55-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
   [junit4]   2> 7436 DEBUG (AutoscalingActionExecutor-55-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger after 100ms
   [junit4]   2> 7436 DEBUG (AutoscalingActionExecutor-55-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing took 55 ms for event 
id=37d9800a794eaT9hcv6ggjwa2gt0u7yah8t3xij
   [junit4]   2> 7436 DEBUG (simCloudManagerPool-53-thread-11) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 7437 DEBUG (simCloudManagerPool-53-thread-11) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
4
   [junit4]   2> 7437 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] 
o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 7437 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] 
o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 7437 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] 
o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
   [junit4]   2> 7478 INFO  (Simulated OverseerAutoScalingTriggerThread) [    ] 
o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are 
closed, exiting.
   [junit4]   2> 7478 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeAddedTrigger-seed#[4444F3BA3046D308]) [ 
   ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor 
now
   [junit4]   2> 7478 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeAddedTrigger-seed#[4444F3BA3046D308]) [ 
   ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 7478 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeAddedTrigger-seed#[4444F3BA3046D308]) [ 
   ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 7478 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeAddedTrigger-seed#[4444F3BA3046D308]) [ 
   ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool 
executor
   [junit4]   2> 7478 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeAddedTrigger-seed#[4444F3BA3046D308]) [ 
   ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 7478 DEBUG 
(TEST-TestSimTriggerIntegration.testNodeAddedTrigger-seed#[4444F3BA3046D308]) [ 
   ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed 
explicitly
   [junit4]   2> 7479 INFO  
(TEST-TestSimTriggerIntegration.testNodeAddedTrigger-seed#[4444F3BA3046D308]) [ 
   ] o.a.s.SolrTestCaseJ4 ###Ending testNodeAddedTrigger
   [junit4] OK      0.23s J1 | TestSimTriggerIntegration.testNodeAddedTrigger
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.sim.TestSimTriggerIntegration_4444F3BA3046D308-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, 
docValues:{}, maxPointsInLeafNode=494, maxMBSortInHeap=5.063057871809752, 
sim=RandomSimilarity(queryNorm=true): {}, locale=en-SE, timezone=Asia/Seoul
   [junit4]   2> NOTE: Linux 4.15.0-42-generic amd64/Oracle Corporation 11 
(64-bit)/cpus=8,threads=1,free=442474504,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [TestSimTriggerIntegration]
   [junit4] Completed [3/5 (3!)] on J1 in 6.89s, 11 tests, 1 failure, 3 skipped 
<<< FAILURES!

[...truncated 47 lines...]
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/common-build.xml:1572: The 
following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/common-build.xml:1099: 
There were test failures: 5 suites, 55 tests, 3 failures, 15 ignored (15 
assumptions) [seed: 4444F3BA3046D308]

Total time: 12 seconds

[repro] Setting last failure code to 256

[repro] Failures:
[repro]   3/5 failed: 
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration
[repro] Exiting with code 256
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Archiving artifacts
Setting 
ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting 
ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all 
warnings.
<Git Blamer> GIT_COMMIT=9aa15a2accd47214dc2d76a035fa31450a079f62, 
workspace=/var/lib/jenkins/workspace/Lucene-Solr-7.x-Linux
[WARNINGS] Computing warning deltas based on reference build #3295
Recording test results
Setting 
ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting 
ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org