Build: https://ci-builds.apache.org/job/Lucene/job/Lucene-Solr-Tests-8.x/705/
1 tests failed.
FAILED:
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testTriggerThrottling
Error Message:
Both triggers did not fire event after await()ing an excessive amount of time
Stack Trace:
java.lang.AssertionError: Both triggers did not fire event after await()ing an
excessive amount of time
at
__randomizedtesting.SeedInfo.seed([B760B5AF094D5B03:4C421D8ADBE7B891]:0)
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.assertTrue(Assert.java:42)
at
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testTriggerThrottling(TestSimTriggerIntegration.java:197)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Build Log:
[...truncated 13426 lines...]
[junit4] Suite:
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration
[junit4] 2> 141103 INFO
(SUITE-TestSimTriggerIntegration-seed#[B760B5AF094D5B03]-worker) [ ]
o.a.s.SolrTestCase Setting 'solr.default.confdir' system property to
test-framework derived value of
'/home/jenkins/jenkins-slave/workspace/Lucene/Lucene-Solr-Tests-8.x/solr/server/solr/configsets/_default/conf'
[junit4] 2> 141104 INFO
(SUITE-TestSimTriggerIntegration-seed#[B760B5AF094D5B03]-worker) [ ]
o.a.s.SolrTestCaseJ4 Created dataDir:
/home/jenkins/jenkins-slave/workspace/Lucene/Lucene-Solr-Tests-8.x/solr/build/solr-core/test/J3/temp/solr.cloud.autoscaling.sim.TestSimTriggerIntegration_B760B5AF094D5B03-001/data-dir-22-001
[junit4] 2> 141104 WARN
(SUITE-TestSimTriggerIntegration-seed#[B760B5AF094D5B03]-worker) [ ]
o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=13 numCloses=13
[junit4] 2> 141104 INFO
(SUITE-TestSimTriggerIntegration-seed#[B760B5AF094D5B03]-worker) [ ]
o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false)
w/NUMERIC_DOCVALUES_SYSPROP=true
[junit4] 2> 141106 INFO
(SUITE-TestSimTriggerIntegration-seed#[B760B5AF094D5B03]-worker) [ ]
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via:
@org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
[junit4] 2> 141106 INFO
(SUITE-TestSimTriggerIntegration-seed#[B760B5AF094D5B03]-worker) [ ]
o.a.s.SolrTestCaseJ4 SecureRandom sanity checks:
test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 141109 INFO
(TEST-TestSimTriggerIntegration.testNodeAddedTrigger-seed#[B760B5AF094D5B03]) [
] o.a.s.SolrTestCaseJ4 ###Starting testNodeAddedTrigger
[junit4] 2> 141143 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and
.scheduled_maintenance triggers
[junit4] 2> 141143 DEBUG
(TEST-TestSimTriggerIntegration.testNodeAddedTrigger-seed#[B760B5AF094D5B03]) [
] o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader:
127.0.0.1:10000_solr
[junit4] 2> 141143 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode
version 1
[junit4] 2> 141143 DEBUG
(TEST-TestSimTriggerIntegration.testNodeAddedTrigger-seed#[B760B5AF094D5B03]) [
] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states,
currentVersion=0
[junit4] 2> 141144 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 0
[junit4] 2> 141144 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
[junit4] 2> 141144 DEBUG
(TEST-TestSimTriggerIntegration.testNodeAddedTrigger-seed#[B760B5AF094D5B03]) [
] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 0
[junit4] 2> 141144 INFO
(TEST-TestSimTriggerIntegration.testNodeAddedTrigger-seed#[B760B5AF094D5B03]) [
] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics
history in memory.
[junit4] 2> 141147 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial
livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
[junit4] 2> 141148 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread -- deactivating old nodeLost / nodeAdded
markers
[junit4] 2> 141151 DEBUG (simCloudManagerPool-1116-thread-2) [ ]
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 141152 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
[junit4] 2> 141152 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 141152 DEBUG (simCloudManagerPool-1116-thread-2) [ ]
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version
2
[junit4] 2> 141158 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
[junit4] 2> 141162 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread -- deactivating old nodeLost / nodeAdded
markers
[junit4] 2> 141162 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
[junit4] 2> 141168 DEBUG (simCloudManagerPool-1116-thread-3) [ ]
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 141168 DEBUG (simCloudManagerPool-1116-thread-3) [ ]
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version
3
[junit4] 2> 141168 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
[junit4] 2> 141169 INFO (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.s.TestSimTriggerIntegration TestTriggerAction init
[junit4] 2> 141169 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger - Initial
livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
[junit4] 2> 141169 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated
with properties: {event=nodeAdded, waitFor=1, actions=[{name=test,
class=org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration$TestTriggerAction}],
enabled=true}
[junit4] 2> 141169 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread -- deactivating old nodeLost / nodeAdded
markers
[junit4] 2> 141169 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
[junit4] 2> 141169 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 141169 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 141169 DEBUG
(TEST-TestSimTriggerIntegration.testNodeAddedTrigger-seed#[B760B5AF094D5B03]) [
] o.a.s.c.a.s.SimClusterStateProvider -- creating marker:
/autoscaling/nodeAdded/127.0.0.1:10002_solr
[junit4] 2> 141172 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 2
[junit4] 2> 141190 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 141190 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 141190 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10002_solr at time
5528806642814031
[junit4] 2> 141192 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 141210 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 141210 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 141211 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing
registered processor for nodes: [127.0.0.1:10002_solr] added at times
[5528806642814031], now=5528807662251881
[junit4] 2> 141212 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"13a46b9bfa0c4fTv4mhjt1g0v5c5lfmh1r2rn1b",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":5528806642814031,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[5528806642814031],
[junit4] 2> "preferredOperation":"movereplica",
[junit4] 2> "nodeNames":["127.0.0.1:10002_solr"],
[junit4] 2> "replicaType":"NRT"}}
[junit4] 2> 141212 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas,
node_added_trigger]
[junit4] 2> 141212 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.SystemLogListener Collection .system missing, skip sending event {
[junit4] 2> "id":"13a46b9bfa0c4fTv4mhjt1g0v5c5lfmh1r2rn1b",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":5528806642814031,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[5528806642814031],
[junit4] 2> "preferredOperation":"movereplica",
[junit4] 2> "_enqueue_time_":5528807746781881,
[junit4] 2> "nodeNames":["127.0.0.1:10002_solr"],
[junit4] 2> "replicaType":"NRT"}}
[junit4] 2> 141214 DEBUG (AutoscalingActionExecutor-1118-thread-1) [ ]
o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"13a46b9bfa0c4fTv4mhjt1g0v5c5lfmh1r2rn1b",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":5528806642814031,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[5528806642814031],
[junit4] 2> "preferredOperation":"movereplica",
[junit4] 2> "_enqueue_time_":5528807746781881,
[junit4] 2> "nodeNames":["127.0.0.1:10002_solr"],
[junit4] 2> "replicaType":"NRT"}}
[junit4] 2> 141214 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Removing new node from tracking: 127.0.0.1:10002_solr
[junit4] 2> 141215 DEBUG (AutoscalingActionExecutor-1118-thread-1) [ ]
o.a.s.c.a.SystemLogListener Collection .system missing, skip sending event {
[junit4] 2> "id":"13a46b9bfa0c4fTv4mhjt1g0v5c5lfmh1r2rn1b",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":5528806642814031,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[5528806642814031],
[junit4] 2> "preferredOperation":"movereplica",
[junit4] 2> "_enqueue_time_":5528807746781881,
[junit4] 2> "nodeNames":["127.0.0.1:10002_solr"],
[junit4] 2> "replicaType":"NRT"}}
[junit4] 2> 141215 DEBUG (AutoscalingActionExecutor-1118-thread-1) [ ]
o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
[junit4] 2> 141215 DEBUG (AutoscalingActionExecutor-1118-thread-1) [ ]
o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger after 100ms
[junit4] 2> 141215 DEBUG (AutoscalingActionExecutor-1118-thread-1) [ ]
o.a.s.c.a.ScheduledTriggers -- processing took 65 ms for event
id=13a46b9bfa0c4fTv4mhjt1g0v5c5lfmh1r2rn1b
[junit4] 2> 141217 DEBUG (simCloudManagerPool-1116-thread-4) [ ]
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 141217 DEBUG (simCloudManagerPool-1116-thread-4) [ ]
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version
4
[junit4] 2> 141218 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
[junit4] 2> 141218 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread -- deactivating old nodeLost / nodeAdded
markers
[junit4] 2> 141218 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
[junit4] 2> 141315 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 141315 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 141315 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 141336 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 141336 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 141336 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 141356 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 141356 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 141356 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 141376 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 141376 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 141376 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 141396 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 141396 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 141396 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 141416 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 141416 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 141416 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 141437 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 141437 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 141437 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 141457 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 141457 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 141457 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 141477 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 141477 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 141477 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 141497 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 141497 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 141497 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 141518 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 141518 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 141518 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 141538 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 141538 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 141538 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 141558 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 141558 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 141558 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 141578 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 141578 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 141578 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 141598 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 141599 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 141599 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 141619 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 141619 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 141619 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 141639 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 141639 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 141639 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 141659 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 141659 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 141659 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 141679 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 141679 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 141680 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 141700 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 141700 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 141700 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 141720 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 141720 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 141720 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 141740 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 141740 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 141740 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 141761 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 141761 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 141761 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 141781 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 141781 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 141781 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 141801 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 141801 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 141801 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 141822 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 141822 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 141822 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 141842 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 141842 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 141842 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 141862 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 141862 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 141862 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 141882 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 141882 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 141882 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 141903 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 141903 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 141903 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 141923 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 141923 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 141923 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 141943 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 141943 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 141943 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 141964 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 141964 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 141964 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 141984 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 141984 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 141984 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142004 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142004 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142004 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142025 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142025 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142025 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142045 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142045 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142045 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142065 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142065 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142065 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142085 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142085 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142086 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142106 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142106 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142106 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142126 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142126 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142126 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142146 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142146 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142146 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142167 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142167 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142167 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142187 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142187 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142187 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142207 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142207 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142207 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142228 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142228 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142228 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142248 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142248 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142248 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142268 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142268 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142268 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142288 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142288 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142289 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142309 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142309 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142309 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142329 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142329 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142329 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142349 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142349 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142349 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142369 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142369 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142369 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142389 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142389 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142390 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142410 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142410 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142410 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142430 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142430 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142430 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142450 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142450 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142450 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142470 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142470 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142470 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142491 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142491 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142491 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142511 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142511 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142511 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142531 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142531 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142531 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142551 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142551 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142551 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142571 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142571 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142571 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142591 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142591 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142592 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142612 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142612 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142612 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142632 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142632 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142632 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142652 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142652 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142652 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142672 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142672 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142672 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142692 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142693 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142693 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142713 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142713 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142713 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142733 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142733 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142733 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142753 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142753 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142753 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142773 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142773 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142773 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142793 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142793 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142793 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142814 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142814 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142814 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142834 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142834 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142834 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142854 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142854 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142854 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142874 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142874 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142874 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142894 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142894 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142894 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142915 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142915 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142915 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142935 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142935 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142935 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142955 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142955 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142955 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142975 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142975 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142975 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 142995 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 142995 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 142995 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143016 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143016 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143016 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143036 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143036 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143036 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143056 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143056 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143056 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143076 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143076 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143076 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143096 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143096 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143096 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143117 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143117 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143117 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143137 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143137 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143137 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143157 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143157 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143157 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143177 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143177 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143177 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143197 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143197 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143198 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143218 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143218 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143218 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143238 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143238 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143238 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143258 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143258 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143258 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143278 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143278 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143278 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143298 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143298 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143298 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143318 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143318 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143319 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143339 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143339 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143339 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143359 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143359 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143359 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143379 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143379 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143379 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143399 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143399 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143399 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143419 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143419 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143420 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143440 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143440 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143440 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143460 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143460 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143460 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143480 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143480 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143480 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143500 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143500 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143501 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143521 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143521 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143521 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143541 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143541 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143541 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143561 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143561 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143561 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143581 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143581 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143581 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143602 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143602 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143602 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143622 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143622 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143622 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143642 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143642 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143642 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143662 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143662 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143662 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143682 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143682 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143682 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143702 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143702 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143702 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143723 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143723 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143723 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143743 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143743 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143743 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143763 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143763 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143763 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143783 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143783 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143783 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143803 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143803 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143803 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143824 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143824 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143824 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143844 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143844 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143844 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143864 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143864 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143864 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143884 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143884 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143884 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143904 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143904 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143904 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143924 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143924 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143924 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143945 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143945 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143945 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143965 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143965 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143965 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 143985 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 143985 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 143985 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 144005 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 144005 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 144005 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 144025 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 144025 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 144025 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 144046 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 144046 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 144046 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 144066 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 144066 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 144066 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 144086 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 144086 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 144086 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 144106 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 144106 DEBUG (ScheduledTrigger-1117-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 144106 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 144127 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 144127 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 144127 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 144147 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 144147 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 144147 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 144167 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 144167 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 144167 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 144187 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 144187 DEBUG (ScheduledTrigger-1117-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 144187 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 144207 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 144207 DEBUG (ScheduledTrigger-1117-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 144207 DEBUG (ScheduledTrigger-1117-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 144219 INFO (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are
closed, exiting.
[junit4] 2> 144219 DEBUG
(TEST-TestSimTriggerIntegration.testNodeAddedTrigger-seed#[B760B5AF094D5B03]) [
] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor
now
[junit4] 2> 144219 DEBUG
(TEST-TestSimTriggerIntegration.testNodeAddedTrigger-seed#[B760B5AF094D5B03]) [
] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
[junit4] 2> 144219 DEBUG
(TEST-TestSimTriggerIntegration.testNodeAddedTrigger-seed#[B760B5AF094D5B03]) [
] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
[junit4] 2> 144219 DEBUG
(TEST-TestSimTriggerIntegration.testNodeAddedTrigger-seed#[B760B5AF094D5B03]) [
] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread
pool executor
[junit4] 2> 144219 DEBUG
(TEST-TestSimTriggerIntegration.testNodeAddedTrigger-seed#[B760B5AF094D5B03]) [
] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
[junit4] 2> 144219 DEBUG
(TEST-TestSimTriggerIntegration.testNodeAddedTrigger-seed#[B760B5AF094D5B03]) [
] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed
explicitly
[junit4] 2> 144221 INFO
(TEST-TestSimTriggerIntegration.testNodeAddedTrigger-seed#[B760B5AF094D5B03]) [
] o.a.s.SolrTestCaseJ4 ###Ending testNodeAddedTrigger
[junit4] 2> 144224 INFO
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[B760B5AF094D5B03])
[ ] o.a.s.SolrTestCaseJ4 ###Starting testNodeLostTriggerRestoreState
[junit4] 2> 144230 DEBUG
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[B760B5AF094D5B03])
[ ] o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader:
127.0.0.1:10003_solr
[junit4] 2> 144230 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and
.scheduled_maintenance triggers
[junit4] 2> 144230 DEBUG
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[B760B5AF094D5B03])
[ ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states,
currentVersion=0
[junit4] 2> 144230 DEBUG
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[B760B5AF094D5B03])
[ ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 0
[junit4] 2> 144230 INFO
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[B760B5AF094D5B03])
[ ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics
history in memory.
[junit4] 2> 144230 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode
version 1
[junit4] 2> 144231 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 0
[junit4] 2> 144231 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
[junit4] 2> 144231 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial
livenodes: [127.0.0.1:10004_solr, 127.0.0.1:10003_solr]
[junit4] 2> 144231 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread -- deactivating old nodeLost / nodeAdded
markers
[junit4] 2> 144231 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
[junit4] 2> 144231 DEBUG (ScheduledTrigger-1123-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 144232 DEBUG (simCloudManagerPool-1122-thread-2) [ ]
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 144232 DEBUG (simCloudManagerPool-1122-thread-2) [ ]
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version
2
[junit4] 2> 144233 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
[junit4] 2> 144233 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread -- deactivating old nodeLost / nodeAdded
markers
[junit4] 2> 144233 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
[junit4] 2> 144234 DEBUG (simCloudManagerPool-1122-thread-3) [ ]
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 144234 DEBUG (simCloudManagerPool-1122-thread-3) [ ]
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version
3
[junit4] 2> 144235 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
[junit4] 2> 144235 INFO (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.s.TestSimTriggerIntegration TestTriggerAction init
[junit4] 2> 144235 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.NodeLostTrigger NodeLostTrigger node_lost_restore_trigger -
Initial livenodes: [127.0.0.1:10004_solr, 127.0.0.1:10003_solr]
[junit4] 2> 144235 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread -- deactivating old nodeLost / nodeAdded
markers
[junit4] 2> 144235 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
[junit4] 2> 144235 DEBUG (ScheduledTrigger-1123-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 144252 DEBUG (ScheduledTrigger-1123-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 2
[junit4] 2> 144255 DEBUG (ScheduledTrigger-1123-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 3 and last live nodes: 2
[junit4] 2> 144257 DEBUG
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[B760B5AF094D5B03])
[ ] o.a.s.c.a.s.SimClusterStateProvider -- created marker:
/autoscaling/nodeLost/127.0.0.1:10005_solr
[junit4] 2> 144272 DEBUG (ScheduledTrigger-1123-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 3
[junit4] 2> 144272 DEBUG (ScheduledTrigger-1123-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10005_solr
[junit4] 2> 144275 DEBUG (ScheduledTrigger-1123-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 3
[junit4] 2> 144275 DEBUG (ScheduledTrigger-1123-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10005_solr
[junit4] 2> 144278 DEBUG (simCloudManagerPool-1122-thread-4) [ ]
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 144278 DEBUG (simCloudManagerPool-1122-thread-4) [ ]
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version
4
[junit4] 2> 144279 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
[junit4] 2> 144279 INFO (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.s.TestSimTriggerIntegration TestTriggerAction init
[junit4] 2> 144279 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.NodeLostTrigger NodeLostTrigger node_lost_resto
[...truncated too long message...]
s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with
properties: {event=nodeAdded, waitFor=10, actions=[{name=test,
class=org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration$TestTriggerAction}],
enabled=true}
[junit4] 2> 209034 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.NodeAddedTrigger Adding node from marker path: 127.0.0.1:10033_solr
[junit4] 2> 209034 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.NodeAddedTrigger Adding node from marker path: 127.0.0.1:10034_solr
[junit4] 2> 209035 DEBUG (ScheduledTrigger-1188-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 4 and last live nodes: 4
[junit4] 2> 209035 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread -- deactivating old nodeLost / nodeAdded
markers
[junit4] 2> 209035 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
[junit4] 2> 209035 DEBUG (ScheduledTrigger-1188-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 209035 DEBUG (ScheduledTrigger-1188-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 209055 DEBUG (ScheduledTrigger-1188-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 4 and last live nodes: 4
[junit4] 2> 209055 DEBUG (ScheduledTrigger-1188-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 209055 DEBUG (ScheduledTrigger-1188-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 209075 DEBUG (ScheduledTrigger-1188-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 4 and last live nodes: 4
[junit4] 2> 209075 DEBUG (ScheduledTrigger-1188-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 209075 DEBUG (ScheduledTrigger-1188-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 209096 DEBUG (ScheduledTrigger-1188-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 4 and last live nodes: 4
[junit4] 2> 209096 DEBUG (ScheduledTrigger-1188-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 209096 DEBUG (ScheduledTrigger-1188-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 209116 DEBUG (ScheduledTrigger-1188-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 209116 DEBUG (ScheduledTrigger-1188-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 4 and last live nodes: 4
[junit4] 2> 209117 DEBUG (ScheduledTrigger-1188-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 209137 DEBUG (ScheduledTrigger-1188-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 209137 DEBUG (ScheduledTrigger-1188-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 4 and last live nodes: 4
[junit4] 2> 209137 DEBUG (ScheduledTrigger-1188-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 209137 DEBUG (ScheduledTrigger-1188-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing
registered processor for nodes: [127.0.0.1:10034_solr] added at times
[5532193723620281], now=5532203979651631
[junit4] 2> 209137 DEBUG (ScheduledTrigger-1188-thread-3) [ ]
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"13a78039b99bb9Tv4mhjt1g0v5c5lfmh1r2rn22",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":5532193723620281,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[5532193723620281],
[junit4] 2> "preferredOperation":"movereplica",
[junit4] 2> "nodeNames":["127.0.0.1:10034_solr"],
[junit4] 2> "replicaType":"NRT"}}
[junit4] 2> 209137 DEBUG (ScheduledTrigger-1188-thread-3) [ ]
o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas,
node_added_trigger]
[junit4] 2> 209137 DEBUG (ScheduledTrigger-1188-thread-3) [ ]
o.a.s.c.a.SystemLogListener Collection .system missing, skip sending event {
[junit4] 2> "id":"13a78039b99bb9Tv4mhjt1g0v5c5lfmh1r2rn22",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":5532193723620281,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[5532193723620281],
[junit4] 2> "preferredOperation":"movereplica",
[junit4] 2> "_enqueue_time_":5532204000402681,
[junit4] 2> "nodeNames":["127.0.0.1:10034_solr"],
[junit4] 2> "replicaType":"NRT"}}
[junit4] 2> 209138 DEBUG (ScheduledTrigger-1188-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Removing new node from tracking: 127.0.0.1:10034_solr
[junit4] 2> 209138 DEBUG (AutoscalingActionExecutor-1189-thread-1) [ ]
o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"13a78039b99bb9Tv4mhjt1g0v5c5lfmh1r2rn22",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":5532193723620281,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[5532193723620281],
[junit4] 2> "preferredOperation":"movereplica",
[junit4] 2> "_enqueue_time_":5532204000402681,
[junit4] 2> "nodeNames":["127.0.0.1:10034_solr"],
[junit4] 2> "replicaType":"NRT"}}
[junit4] 2> 209138 DEBUG (AutoscalingActionExecutor-1189-thread-1) [ ]
o.a.s.c.a.SystemLogListener Collection .system missing, skip sending event {
[junit4] 2> "id":"13a78039b99bb9Tv4mhjt1g0v5c5lfmh1r2rn22",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":5532193723620281,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[5532193723620281],
[junit4] 2> "preferredOperation":"movereplica",
[junit4] 2> "_enqueue_time_":5532204000402681,
[junit4] 2> "nodeNames":["127.0.0.1:10034_solr"],
[junit4] 2> "replicaType":"NRT"}}
[junit4] 2> 209138 DEBUG (AutoscalingActionExecutor-1189-thread-1) [ ]
o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger after 100ms
[junit4] 2> 209138 DEBUG (AutoscalingActionExecutor-1189-thread-1) [ ]
o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
[junit4] 2> 209138 DEBUG (AutoscalingActionExecutor-1189-thread-1) [ ]
o.a.s.c.a.ScheduledTriggers -- processing took 14 ms for event
id=13a78039b99bb9Tv4mhjt1g0v5c5lfmh1r2rn22
[junit4] 2> 209138 INFO (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are
closed, exiting.
[junit4] 2> 209138 DEBUG
(TEST-TestSimTriggerIntegration.testEventFromRestoredState-seed#[B760B5AF094D5B03])
[ ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool
executor now
[junit4] 2> 209139 DEBUG
(TEST-TestSimTriggerIntegration.testEventFromRestoredState-seed#[B760B5AF094D5B03])
[ ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
[junit4] 2> 209139 DEBUG
(TEST-TestSimTriggerIntegration.testEventFromRestoredState-seed#[B760B5AF094D5B03])
[ ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
[junit4] 2> 209139 DEBUG
(TEST-TestSimTriggerIntegration.testEventFromRestoredState-seed#[B760B5AF094D5B03])
[ ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread
pool executor
[junit4] 2> 209139 DEBUG
(TEST-TestSimTriggerIntegration.testEventFromRestoredState-seed#[B760B5AF094D5B03])
[ ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
[junit4] 2> 209139 DEBUG
(TEST-TestSimTriggerIntegration.testEventFromRestoredState-seed#[B760B5AF094D5B03])
[ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed
explicitly
[junit4] 2> 209139 INFO
(TEST-TestSimTriggerIntegration.testEventFromRestoredState-seed#[B760B5AF094D5B03])
[ ] o.a.s.SolrTestCaseJ4 ###Ending testEventFromRestoredState
[junit4] 2> NOTE: leaving temporary files on disk at:
/home/jenkins/jenkins-slave/workspace/Lucene/Lucene-Solr-Tests-8.x/solr/build/solr-core/test/J3/temp/solr.cloud.autoscaling.sim.TestSimTriggerIntegration_B760B5AF094D5B03-001
[junit4] 2> NOTE: test params are: codec=CheapBastard,
sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=ca,
timezone=SystemV/HST10
[junit4] 2> NOTE: Linux 5.4.0-45-generic amd64/Oracle Corporation
1.8.0_252 (64-bit)/cpus=4,threads=1,free=133597784,total=427819008
[junit4] 2> NOTE: All tests run in this JVM: [MetricsConfigTest,
PingRequestHandlerTest, TestExpandComponent, AutoAddReplicasPlanActionTest,
TestDocBasedVersionConstraints, PropertiesRequestHandlerTest, UtilsToolTest,
TestConfigSetsAPIShareSchema, TestDynamicFieldCollectionResource,
TestUninvertingReader, TestSlowCompositeReaderWrapper,
LegacyCloudClusterPropTest, MultiDestinationAuditLoggerTest,
InfixSuggestersTest, TestSystemIdResolver, DeleteStatusTest,
SignatureUpdateProcessorFactoryTest, TestCSVLoader, HdfsSyncSliceTest,
LukeRequestHandlerTest, TestSimTriggerIntegration]
[junit4] Completed [69/921 (1!)] on J3 in 68.29s, 11 tests, 1 failure <<<
FAILURES!
[...truncated 53559 lines...]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
