Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/5004/
Java: 64bit/jdk-9 -XX:+UseCompressedOops -XX:+UseG1GC
1 tests failed.
FAILED:
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testNodeLostTriggerRestoreState
Error Message:
The trigger did not fire at all
Stack Trace:
java.lang.AssertionError: The trigger did not fire at all
at
__randomizedtesting.SeedInfo.seed([313D1D34850DDBA:28EC0488D228C86A]:0)
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testNodeLostTriggerRestoreState(TestSimTriggerIntegration.java:332)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)
Build Log:
[...truncated 13872 lines...]
[junit4] Suite:
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration
[junit4] 2> Creating dataDir:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.sim.TestSimTriggerIntegration_313D1D34850DDBA-001/init-core-data-001
[junit4] 2> 1257023 DEBUG
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[313D1D34850DDBA])
[ ] o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader:
127.0.0.1:10029_solr
[junit4] 2> 1257023 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and
.scheduled_maintenance triggers
[junit4] 2> 1257023 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode
version 1
[junit4] 2> 1257023 DEBUG
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[313D1D34850DDBA])
[ ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states,
currentVersion=0
[junit4] 2> 1257024 DEBUG
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[313D1D34850DDBA])
[ ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 0
[junit4] 2> 1257024 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 0
[junit4] 2> 1257024 INFO
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[313D1D34850DDBA])
[ ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics
history in memory.
[junit4] 2> 1257024 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
[junit4] 2> 1257024 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial
livenodes: [127.0.0.1:10029_solr, 127.0.0.1:10030_solr]
[junit4] 2> 1257024 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
[junit4] 2> 1257024 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
[junit4] 2> 1257025 DEBUG (ScheduledTrigger-8040-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257026 DEBUG (simCloudManagerPool-8039-thread-2) [ ]
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1257026 DEBUG (simCloudManagerPool-8039-thread-2) [ ]
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version
2
[junit4] 2> 1257027 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
[junit4] 2> 1257027 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
[junit4] 2> 1257027 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
[junit4] 2> 1257030 DEBUG (simCloudManagerPool-8039-thread-3) [ ]
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1257030 DEBUG (simCloudManagerPool-8039-thread-3) [ ]
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version
3
[junit4] 2> 1257030 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
[junit4] 2> 1257030 INFO (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.s.TestSimTriggerIntegration TestTriggerAction init
[junit4] 2> 1257030 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.NodeLostTrigger NodeLostTrigger node_lost_restore_trigger -
Initial livenodes: [127.0.0.1:10029_solr, 127.0.0.1:10030_solr]
[junit4] 2> 1257030 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
[junit4] 2> 1257030 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257031 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
[junit4] 2> 1257030 DEBUG
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[313D1D34850DDBA])
[ ] o.a.s.c.a.s.SimClusterStateProvider -- created marker:
/autoscaling/nodeLost/127.0.0.1:10031_solr
[junit4] 2> 1257053 DEBUG (ScheduledTrigger-8040-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257053 DEBUG (ScheduledTrigger-8040-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257072 DEBUG (simCloudManagerPool-8039-thread-4) [ ]
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1257072 DEBUG (simCloudManagerPool-8039-thread-4) [ ]
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version
4
[junit4] 2> 1257072 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
[junit4] 2> 1257072 INFO (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.s.TestSimTriggerIntegration TestTriggerAction init
[junit4] 2> 1257072 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.NodeLostTrigger NodeLostTrigger node_lost_restore_trigger -
Initial livenodes: [127.0.0.1:10029_solr, 127.0.0.1:10030_solr]
[junit4] 2> 1257073 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
[junit4] 2> 1257073 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
[junit4] 2> 1257073 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257073 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257093 DEBUG (ScheduledTrigger-8040-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257093 DEBUG (ScheduledTrigger-8040-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257114 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257114 DEBUG (ScheduledTrigger-8040-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257136 DEBUG (ScheduledTrigger-8040-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257136 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257156 DEBUG (ScheduledTrigger-8040-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257156 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257184 DEBUG (ScheduledTrigger-8040-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257185 DEBUG (ScheduledTrigger-8040-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257212 DEBUG (ScheduledTrigger-8040-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257212 DEBUG (ScheduledTrigger-8040-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257232 DEBUG (ScheduledTrigger-8040-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257232 DEBUG (ScheduledTrigger-8040-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257253 DEBUG (ScheduledTrigger-8040-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257253 DEBUG (ScheduledTrigger-8040-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257283 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257283 DEBUG (ScheduledTrigger-8040-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257313 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257313 DEBUG (ScheduledTrigger-8040-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257336 DEBUG (ScheduledTrigger-8040-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257336 DEBUG (ScheduledTrigger-8040-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257364 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257364 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257384 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257384 DEBUG (ScheduledTrigger-8040-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257405 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257405 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257435 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257435 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257455 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257455 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257481 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257481 DEBUG (ScheduledTrigger-8040-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257506 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257507 DEBUG (ScheduledTrigger-8040-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257537 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257537 DEBUG (ScheduledTrigger-8040-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257561 DEBUG (ScheduledTrigger-8040-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257561 DEBUG (ScheduledTrigger-8040-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257595 DEBUG (ScheduledTrigger-8040-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257595 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257615 DEBUG (ScheduledTrigger-8040-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257615 DEBUG (ScheduledTrigger-8040-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257636 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257636 DEBUG (ScheduledTrigger-8040-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257656 DEBUG (ScheduledTrigger-8040-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257656 DEBUG (ScheduledTrigger-8040-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257677 DEBUG (ScheduledTrigger-8040-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257677 DEBUG (ScheduledTrigger-8040-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257699 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257699 DEBUG (ScheduledTrigger-8040-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257721 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257721 DEBUG (ScheduledTrigger-8040-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257744 DEBUG (ScheduledTrigger-8040-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257744 DEBUG (ScheduledTrigger-8040-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257764 DEBUG (ScheduledTrigger-8040-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257764 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257785 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257785 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257806 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257806 DEBUG (ScheduledTrigger-8040-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257827 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257827 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257850 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257850 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257871 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257871 DEBUG (ScheduledTrigger-8040-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257895 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257895 DEBUG (ScheduledTrigger-8040-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257918 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257918 DEBUG (ScheduledTrigger-8040-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257939 DEBUG (ScheduledTrigger-8040-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257939 DEBUG (ScheduledTrigger-8040-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257959 DEBUG (ScheduledTrigger-8040-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257959 DEBUG (ScheduledTrigger-8040-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257984 DEBUG (ScheduledTrigger-8040-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1257984 DEBUG (ScheduledTrigger-8040-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258004 DEBUG (ScheduledTrigger-8040-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258004 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258027 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258027 DEBUG (ScheduledTrigger-8040-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258047 DEBUG (ScheduledTrigger-8040-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258047 DEBUG (ScheduledTrigger-8040-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258074 DEBUG (ScheduledTrigger-8040-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258074 DEBUG (ScheduledTrigger-8040-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258095 DEBUG (ScheduledTrigger-8040-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258095 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258115 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258115 DEBUG (ScheduledTrigger-8040-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258136 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258136 DEBUG (ScheduledTrigger-8040-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258156 DEBUG (ScheduledTrigger-8040-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258156 DEBUG (ScheduledTrigger-8040-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258182 DEBUG (ScheduledTrigger-8040-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258182 DEBUG (ScheduledTrigger-8040-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258205 DEBUG (ScheduledTrigger-8040-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258205 DEBUG (ScheduledTrigger-8040-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258232 DEBUG (ScheduledTrigger-8040-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258232 DEBUG (ScheduledTrigger-8040-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258254 DEBUG (ScheduledTrigger-8040-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258254 DEBUG (ScheduledTrigger-8040-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258282 DEBUG (ScheduledTrigger-8040-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258282 DEBUG (ScheduledTrigger-8040-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258305 DEBUG (ScheduledTrigger-8040-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258305 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258332 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258332 DEBUG (ScheduledTrigger-8040-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258362 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258362 DEBUG (ScheduledTrigger-8040-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258385 DEBUG (ScheduledTrigger-8040-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258385 DEBUG (ScheduledTrigger-8040-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258410 DEBUG (ScheduledTrigger-8040-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258410 DEBUG (ScheduledTrigger-8040-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258436 DEBUG (ScheduledTrigger-8040-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258436 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258466 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258466 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258491 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258491 DEBUG (ScheduledTrigger-8040-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258511 DEBUG (ScheduledTrigger-8040-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258512 DEBUG (ScheduledTrigger-8040-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258533 DEBUG (ScheduledTrigger-8040-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258533 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258563 DEBUG (ScheduledTrigger-8040-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258563 DEBUG (ScheduledTrigger-8040-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258599 DEBUG (ScheduledTrigger-8040-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258600 DEBUG (ScheduledTrigger-8040-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258630 DEBUG (ScheduledTrigger-8040-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258630 DEBUG (ScheduledTrigger-8040-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258654 DEBUG (ScheduledTrigger-8040-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258654 DEBUG (ScheduledTrigger-8040-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258675 DEBUG (ScheduledTrigger-8040-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258675 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258696 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258696 DEBUG (ScheduledTrigger-8040-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258716 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258717 DEBUG (ScheduledTrigger-8040-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258737 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258737 DEBUG (ScheduledTrigger-8040-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258758 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258758 DEBUG (ScheduledTrigger-8040-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258778 DEBUG (ScheduledTrigger-8040-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258778 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258800 DEBUG (ScheduledTrigger-8040-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258800 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258820 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258820 DEBUG (ScheduledTrigger-8040-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258840 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258840 DEBUG (ScheduledTrigger-8040-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258862 DEBUG (ScheduledTrigger-8040-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258862 DEBUG (ScheduledTrigger-8040-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258877 INFO (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are
closed, exiting.
[junit4] 2> 1258881 DEBUG
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[313D1D34850DDBA])
[ ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool
executor now
[junit4] 2> 1258881 DEBUG
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[313D1D34850DDBA])
[ ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
[junit4] 2> 1258882 DEBUG
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[313D1D34850DDBA])
[ ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
[junit4] 2> 1258882 DEBUG
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[313D1D34850DDBA])
[ ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread
pool executor
[junit4] 2> 1258883 DEBUG
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[313D1D34850DDBA])
[ ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
[junit4] 2> 1258883 DEBUG
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[313D1D34850DDBA])
[ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed
explicitly
[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=TestSimTriggerIntegration
-Dtests.method=testNodeLostTriggerRestoreState -Dtests.seed=313D1D34850DDBA
-Dtests.slow=true -Dtests.locale=sr-Latn-BA -Dtests.timezone=Etc/GMT-10
-Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] FAILURE 1.87s J0 |
TestSimTriggerIntegration.testNodeLostTriggerRestoreState <<<
[junit4] > Throwable #1: java.lang.AssertionError: The trigger did not
fire at all
[junit4] > at
__randomizedtesting.SeedInfo.seed([313D1D34850DDBA:28EC0488D228C86A]:0)
[junit4] > at
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testNodeLostTriggerRestoreState(TestSimTriggerIntegration.java:332)
[junit4] > at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4] > at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[junit4] > at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4] > at
java.base/java.lang.reflect.Method.invoke(Method.java:564)
[junit4] > at java.base/java.lang.Thread.run(Thread.java:844)
[junit4] IGNOR/A 0.00s J0 | TestSimTriggerIntegration.testTriggerThrottling
[junit4] > Assumption #1: 'badapple' test group is disabled
(@BadApple(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";))
[junit4] IGNOR/A 0.00s J0 | TestSimTriggerIntegration.testSearchRate
[junit4] > Assumption #1: 'awaitsfix' test group is disabled
(@AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";))
[junit4] IGNOR/A 0.00s J0 | TestSimTriggerIntegration.testNodeLostTrigger
[junit4] > Assumption #1: 'awaitsfix' test group is disabled
(@AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";))
[junit4] IGNOR/A 0.00s J0 |
TestSimTriggerIntegration.testNodeMarkersRegistration
[junit4] > Assumption #1: 'badapple' test group is disabled
(@BadApple(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";))
[junit4] 2> 1258897 DEBUG
(TEST-TestSimTriggerIntegration.testListeners-seed#[313D1D34850DDBA]) [ ]
o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader:
127.0.0.1:10032_solr
[junit4] 2> 1258897 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and
.scheduled_maintenance triggers
[junit4] 2> 1258897 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode
version 1
[junit4] 2> 1258898 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 0
[junit4] 2> 1258898 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
[junit4] 2> 1258898 DEBUG
(TEST-TestSimTriggerIntegration.testListeners-seed#[313D1D34850DDBA]) [ ]
o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states,
currentVersion=0
[junit4] 2> 1258898 DEBUG
(TEST-TestSimTriggerIntegration.testListeners-seed#[313D1D34850DDBA]) [ ]
o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 0
[junit4] 2> 1258898 INFO
(TEST-TestSimTriggerIntegration.testListeners-seed#[313D1D34850DDBA]) [ ]
o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history
in memory.
[junit4] 2> 1258899 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial
livenodes: [127.0.0.1:10033_solr, 127.0.0.1:10032_solr]
[junit4] 2> 1258900 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
[junit4] 2> 1258900 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
[junit4] 2> 1258900 DEBUG (ScheduledTrigger-8046-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258942 DEBUG (ScheduledTrigger-8046-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258954 DEBUG (simCloudManagerPool-8045-thread-2) [ ]
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1258954 DEBUG (simCloudManagerPool-8045-thread-2) [ ]
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version
2
[junit4] 2> 1258954 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
[junit4] 2> 1258955 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
[junit4] 2> 1258955 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
[junit4] 2> 1258964 DEBUG (ScheduledTrigger-8046-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258967 DEBUG (simCloudManagerPool-8045-thread-3) [ ]
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1258967 DEBUG (simCloudManagerPool-8045-thread-3) [ ]
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version
3
[junit4] 2> 1258967 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
[junit4] 2> 1258968 INFO (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.s.TestSimTriggerIntegration TestTriggerAction init
[junit4] 2> 1258968 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger - Initial
livenodes: [127.0.0.1:10033_solr, 127.0.0.1:10032_solr]
[junit4] 2> 1258968 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated
with properties: {event=nodeAdded, waitFor=2, actions=[{name=test,
class=org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration$TestTriggerAction},
{name=test1,
class=org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration$TestDummyAction}],
enabled=true}
[junit4] 2> 1258968 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
[junit4] 2> 1258968 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
[junit4] 2> 1258968 DEBUG (ScheduledTrigger-8046-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1258968 DEBUG (ScheduledTrigger-8046-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 1258983 DEBUG (simCloudManagerPool-8045-thread-4) [ ]
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1258984 DEBUG (simCloudManagerPool-8045-thread-4) [ ]
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version
4
[junit4] 2> 1258984 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
[junit4] 2> 1258984 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
[junit4] 2> 1258984 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
[junit4] 2> 1258986 DEBUG (simCloudManagerPool-8045-thread-5) [ ]
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1258986 DEBUG (simCloudManagerPool-8045-thread-5) [ ]
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version
5
[junit4] 2> 1258987 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
[junit4] 2> 1258987 DEBUG (ScheduledTrigger-8046-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1258987 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
[junit4] 2> 1258987 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
[junit4] 2> 1258987 DEBUG
(TEST-TestSimTriggerIntegration.testListeners-seed#[313D1D34850DDBA]) [ ]
o.a.s.c.a.s.SimClusterStateProvider -- creating marker:
/autoscaling/nodeAdded/127.0.0.1:10034_solr
[junit4] 2> 1258989 DEBUG (ScheduledTrigger-8046-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1258989 DEBUG (ScheduledTrigger-8046-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1258989 DEBUG (ScheduledTrigger-8046-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10034_solr at time
1680194954843260
[junit4] 2> 1259009 DEBUG (ScheduledTrigger-8046-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 2
[junit4] 2> 1259009 DEBUG (ScheduledTrigger-8046-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1259010 DEBUG (ScheduledTrigger-8046-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1259031 DEBUG (ScheduledTrigger-8046-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1259031 DEBUG (ScheduledTrigger-8046-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1259031 DEBUG (ScheduledTrigger-8046-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1259031 DEBUG (ScheduledTrigger-8046-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing
registered processor for nodes: [127.0.0.1:10034_solr] added at times
[1680194954843260], now=1680197055400210
[junit4] 2> 1259031 DEBUG (ScheduledTrigger-8046-thread-2) [ ]
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"5f820df6fd47cTchhl7e9xm4vo4iih1xzf4t7v3",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":1680194954843260,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1680194954843260],
[junit4] 2> "preferredOperation":"movereplica",
[junit4] 2> "nodeNames":["127.0.0.1:10034_solr"]}}
[junit4] 2> 1259031 DEBUG (ScheduledTrigger-8046-thread-2) [ ]
o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas,
node_added_trigger]
[junit4] 2> 1259032 DEBUG (ScheduledTrigger-8046-thread-2) [ ]
o.a.s.c.a.SystemLogListener Collection .system missing, skip sending event {
[junit4] 2> "id":"5f820df6fd47cTchhl7e9xm4vo4iih1xzf4t7v3",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":1680194954843260,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1680194954843260],
[junit4] 2> "preferredOperation":"movereplica",
[junit4] 2> "_enqueue_time_":1680197077286460,
[junit4] 2> "nodeNames":["127.0.0.1:10034_solr"]}}
[junit4] 2> 1259033 DEBUG (ScheduledTrigger-8046-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Removing new node from tracking: 127.0.0.1:10034_solr
[junit4] 2> 1259033 DEBUG (AutoscalingActionExecutor-8047-thread-1) [ ]
o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"5f820df6fd47cTchhl7e9xm4vo4iih1xzf4t7v3",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":1680194954843260,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1680194954843260],
[junit4] 2> "preferredOperation":"movereplica",
[junit4] 2> "_enqueue_time_":1680197077286460,
[junit4] 2> "nodeNames":["127.0.0.1:10034_solr"]}}
[junit4] 2> 1259036 DEBUG (AutoscalingActionExecutor-8047-thread-1) [ ]
o.a.s.c.a.SystemLogListener Collection .system missing, skip sending event {
[junit4] 2> "id":"5f820df6fd47cTchhl7e9xm4vo4iih1xzf4t7v3",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":1680194954843260,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1680194954843260],
[junit4] 2> "preferredOperation":"movereplica",
[junit4] 2> "_enqueue_time_":1680197077286460,
[junit4] 2> "nodeNames":["127.0.0.1:10034_solr"]}}
[junit4] 2> 1259037 DEBUG (AutoscalingActionExecutor-8047-thread-1) [ ]
o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger after 100ms
[junit4] 2> 1259037 DEBUG (AutoscalingActionExecutor-8047-thread-1) [ ]
o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
[junit4] 2> 1259037 DEBUG (AutoscalingActionExecutor-8047-thread-1) [ ]
o.a.s.c.a.ScheduledTriggers -- processing took 199 ms for event
id=5f820df6fd47cTchhl7e9xm4vo4iih1xzf4t7v3
[junit4] 2> 1259100 DEBUG
(TEST-TestSimTriggerIntegration.testListeners-seed#[313D1D34850DDBA]) [ ]
o.a.s.c.a.s.SimClusterStateProvider -- creating marker:
/autoscaling/nodeAdded/127.0.0.1:10035_solr
[junit4] 2> 1259139 DEBUG (ScheduledTrigger-8046-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1259139 DEBUG (ScheduledTrigger-8046-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 1259139 DEBUG (ScheduledTrigger-8046-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10035_solr at time
1680202434957860
[junit4] 2> 1259140 DEBUG (ScheduledTrigger-8046-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 4 and last live nodes: 3
[junit4] 2> 1259164 DEBUG (ScheduledTrigger-8046-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1259164 DEBUG (ScheduledTrigger-8046-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 1259164 DEBUG (ScheduledTrigger-8046-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 4 and last live nodes: 4
[junit4] 2> 1259189 DEBUG (ScheduledTrigger-8046-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1259189 DEBUG (ScheduledTrigger-8046-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 1259189 DEBUG (ScheduledTrigger-8046-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing
registered processor for nodes: [127.0.0.1:10035_solr] added at times
[1680202434957860], now=1680204943405660
[junit4] 2> 1259189 DEBUG (ScheduledTrigger-8046-thread-1) [ ]
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"5f8229d495224Tchhl7e9xm4vo4iih1xzf4t7v4",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":1680202434957860,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1680202434957860],
[junit4] 2> "preferredOperation":"movereplica",
[junit4] 2> "nodeNames":["127.0.0.1:10035_solr"]}}
[junit4] 2> 1259189 DEBUG (ScheduledTrigger-8046-thread-1) [ ]
o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas,
node_added_trigger]
[junit4] 2> 1259189 DEBUG (ScheduledTrigger-8046-thread-1) [ ]
o.a.s.c.a.SystemLogListener Collection .system missing, skip sending event {
[junit4] 2> "id":"5f8229d495224Tchhl7e9xm4vo4iih1xzf4t7v4",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":1680202434957860,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1680202434957860],
[junit4] 2> "preferredOperation":"movereplica",
[junit4] 2> "_enqueue_time_":1680204953299910,
[junit4] 2> "nodeNames":["127.0.0.1:10035_solr"]}}
[junit4] 2> 1259189 DEBUG (ScheduledTrigger-8046-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Removing new node from tracking: 127.0.0.1:10035_solr
[junit4] 2> 1259189 DEBUG (AutoscalingActionExecutor-8047-thread-1) [ ]
o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"5f8229d495224Tchhl7e9xm4vo4iih1xzf4t7v4",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":1680202434957860,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1680202434957860],
[junit4] 2> "preferredOperation":"movereplica",
[junit4] 2> "_enqueue_time_":1680204953299910,
[junit4] 2> "nodeNames":["127.0.0.1:10035_solr"]}}
[junit4] 2> 1259190 DEBUG (AutoscalingActionExecutor-8047-thread-1) [ ]
o.a.s.c.a.SystemLogListener Collection .system missing, skip sending event {
[junit4] 2> "id":"5f8229d495224Tchhl7e9xm4vo4iih1xzf4t7v4",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":1680202434957860,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1680202434957860],
[junit4] 2> "preferredOperation":"movereplica",
[junit4] 2> "_enqueue_time_":1680204953299910,
[junit4] 2> "nodeNames":["127.0.0.1:10035_solr"]}}
[junit4] 2> 1259190 WARN (AutoscalingActionExecutor-8047-thread-1) [ ]
o.a.s.c.a.ScheduledTriggers Exception executing actions
[junit4] 2> org.apache.solr.cloud.autoscaling.TriggerActionException:
Error processing action for trigger event: {
[junit4] 2> "id":"5f8229d495224Tchhl7e9xm4vo4iih1xzf4t7v4",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":1680202434957860,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1680202434957860],
[junit4] 2> "preferredOperation":"movereplica",
[junit4] 2> "_enqueue_time_":1680204953299910,
[junit4] 2> "nodeNames":["127.0.0.1:10035_solr"]}}
[junit4] 2> at
org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$add$3(ScheduledTriggers.java:327)
[java/:?]
[junit4] 2> at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514) [?:?]
[junit4] 2> at
java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
[junit4] 2> at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
[java/:?]
[junit4] 2> at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
[?:?]
[junit4] 2> at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
[?:?]
[junit4] 2> at java.lang.Thread.run(Thread.java:844) [?:?]
[junit4] 2> Caused by: java.lang.RuntimeException: failure
[junit4] 2> at
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration$TestDummyAction.process(TestSimTriggerIntegration.java:943)
~[test/:?]
[junit4] 2> at
org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$add$3(ScheduledTriggers.java:324)
~[java/:?]
[junit4] 2> ... 6 more
[junit4] 2> 1259191 DEBUG (AutoscalingActionExecutor-8047-thread-1) [ ]
o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
[junit4] 2> 1259191 DEBUG (AutoscalingActionExecutor-8047-thread-1) [ ]
o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger after 100ms
[junit4] 2> 1259191 DEBUG (AutoscalingActionExecutor-8047-thread-1) [ ]
o.a.s.c.a.ScheduledTriggers -- processing took 76 ms for event
id=5f8229d495224Tchhl7e9xm4vo4iih1xzf4t7v4
[junit4] 2> 1259257 DEBUG
(TEST-TestSimTriggerIntegration.testListeners-seed#[313D1D34850DDBA]) [ ]
o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
[junit4] 2> 1259257 INFO (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are
closed, exiting.
[junit4] 2> 1259257 DEBUG
(TEST-TestSimTriggerIntegration.testListeners-seed#[313D1D34850DDBA]) [ ]
o.a.s.c.a.ScheduledTriggers Shutting down action executor now
[junit4] 2> 1259257 DEBUG
(TEST-TestSimTriggerIntegration.testListeners-seed#[313D1D34850DDBA]) [ ]
o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
[junit4] 2> 1259258 DEBUG
(TEST-TestSimTriggerIntegration.testListeners-seed#[313D1D34850DDBA]) [ ]
o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool
executor
[junit4] 2> 1259258 DEBUG
(TEST-TestSimTriggerIntegration.testListeners-seed#[313D1D34850DDBA]) [ ]
o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
[junit4] 2> 1259258 DEBUG
(TEST-TestSimTriggerIntegration.testListeners-seed#[313D1D34850DDBA]) [ ]
o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] IGNOR/A 0.00s J0 | TestSimTriggerIntegration.testEventQueue
[junit4] > Assumption #1: 'awaitsfix' test group is disabled
(@AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";))
[junit4] IGNOR/A 0.00s J0 | TestSimTriggerIntegration.testCooldown
[junit4] > Assumption #1: 'badapple' test group is disabled
(@BadApple(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";))
[junit4] IGNOR/A 0.00s J0 |
TestSimTriggerIntegration.testEventFromRestoredState
[junit4] > Assumption #1: 'badapple' test group is disabled
(@BadApple(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";))
[junit4] IGNOR/A 0.00s J0 | TestSimTriggerIntegration.testNodeAddedTrigger
[junit4] > Assumption #1: 'badapple' test group is disabled
(@BadApple(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";))
[junit4] IGNOR/A 0.00s J0 |
TestSimTriggerIntegration.testNodeAddedTriggerRestoreState
[junit4] > Assumption #1: 'badapple' test group is disabled
(@BadApple(bugUrl="https://issues.apache.org/jira/browse/SOLR-12028";))
[junit4] 2> NOTE: leaving temporary files on disk at:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.sim.TestSimTriggerIntegration_313D1D34850DDBA-001
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene80): {},
docValues:{}, maxPointsInLeafNode=1888, maxMBSortInHeap=5.613043991803965,
sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@26f90730),
locale=sr-Latn-BA, timezone=Etc/GMT-10
[junit4] 2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 9
(64-bit)/cpus=3,threads=1,free=427830736,total=536870912
[junit4] 2> NOTE: All tests run in this JVM: [SpellingQueryConverterTest,
TokenizerChainTest, TestDynamicFieldResource, TestNestedDocsSort,
MoveReplicaHDFSFailoverTest, TestFuzzyAnalyzedSuggestions, ResponseBuilderTest,
SolrCoreTest, TestRequestForwarding, DistributedQueryElevationComponentTest,
OutputWriterTest, TestDFISimilarityFactory, TestIBSimilarityFactory,
DefaultValueUpdateProcessorTest, TestSolrDeletionPolicy2,
TriggerEventQueueTest, CoreMergeIndexesAdminHandlerTest, TestDistribIDF,
TestAtomicUpdateErrorCases, QueryResultKeyTest, TestCloudSchemaless,
SolrMetricsIntegrationTest, AdminHandlersProxyTest, TestRawTransformer,
ZkCLITest, TestConfig, TestSolrCloudSnapshots, FacetPivotSmallTest,
AnalyticsQueryTest, FileUtilsTest, ZkFailoverTest, TestComponentsName,
TestCrossCoreJoin, TestMinMaxOnMultiValuedField,
HdfsWriteToMultipleCollectionsTest, SignatureUpdateProcessorFactoryTest,
TestManagedSchemaThreadSafety, SchemaApiFailureTest, ShardRoutingTest,
ShowFileRequestHandlerTest, CleanupOldIndexTest, TestExceedMaxTermLength,
TestGeoJSONResponseWriter, TestChildDocTransformerHierarchy,
OverseerCollectionConfigSetProcessorTest, TestQueryTypes,
TestExtendedDismaxParser, ProtectedTermFilterFactoryTest, TestCollectionAPIs,
DocExpirationUpdateProcessorFactoryTest, TestBlobHandler, TestDocTermOrds,
TestTlogReplica, TestNonDefinedSimilarityFactory, BasicDistributedZk2Test,
Tagger2Test, TestUtils, ChaosMonkeySafeLeaderWithPullReplicasTest,
TestStandardQParsers, FullHLLTest, TestCryptoKeys, TestConfigSetImmutable,
TestConfigSets, BlockDirectoryTest, VersionInfoTest, TestSolrConfigHandler,
SearchRateTriggerTest, SpellCheckCollatorWithCollapseTest,
URLClassifyProcessorTest, TestSolr4Spatial2,
DistributedFacetPivotSmallAdvancedTest, TestFieldTypeResource,
ReplicationFactorTest, GraphQueryTest, PKIAuthenticationIntegrationTest,
CdcrWithNodesRestartsTest, TestRandomFaceting, TestBadConfig,
TestCloudJSONFacetJoinDomain, DeleteNodeTest, DistributedFacetPivotLargeTest,
SortSpecParsingTest, TestSortByMinMaxFunction,
TestTolerantUpdateProcessorRandomCloud, CursorMarkTest, TestLegacyFieldCache,
TestSha256AuthenticationProvider, EmbeddedSolrNoSerializeTest,
TestManagedResource, DebugComponentTest, TestInitParams, RequestHandlersTest,
TestTrie, TestManagedSchema, TestSolrCloudWithDelegationTokens,
CollectionsAPISolrJTest, TestUniqueKeyFieldResource,
ChaosMonkeyNothingIsSafeWithPullReplicasTest, HdfsDirectoryTest,
TestCoreDiscovery, BooleanFieldTest, TestDeprecatedFilters,
LeaderElectionIntegrationTest, TestSimComputePlanAction, TestReplicaProperties,
MBeansHandlerTest, HdfsBasicDistributedZk2Test,
TestCopyFieldCollectionResource, HdfsChaosMonkeyNothingIsSafeTest,
BadCopyFieldTest, TestLazyCores, UpdateLogTest, BJQParserTest,
LoggingHandlerTest, CollectionTooManyReplicasTest, TestSimNodeLostTrigger,
TestJoin, TestIndexSearcher, TestRestManager, TestSlowCompositeReaderWrapper,
BasicZkTest, TestSolrCloudWithSecureImpersonation, TestFieldResource,
TestSolrCoreProperties, TestClusterProperties, TestIntervalFaceting,
ZkStateWriterTest, DistributedExpandComponentTest, TestMultiWordSynonyms,
OutOfBoxZkACLAndCredentialsProvidersTest, MetricsHistoryHandlerTest,
SplitShardTest, TermsComponentTest, TestEmbeddedSolrServerAdminHandler,
DistribJoinFromCollectionTest, DocValuesMissingTest, NotRequiredUniqueKeyTest,
IgnoreCommitOptimizeUpdateProcessorFactoryTest, AssignTest,
DisMaxRequestHandlerTest, TestGraphTermsQParserPlugin,
TestObjectReleaseTracker, TestBM25SimilarityFactory, ReturnFieldsTest,
CdcrRequestHandlerTest, ScheduledTriggerTest, OpenCloseCoreStressTest,
TestConfigSetsAPIExclusivity, TestNamedUpdateProcessors, OverseerTaskQueueTest,
TestSmileRequest, DirectSolrConnectionTest, MetricsConfigTest,
TestSearcherReuse, TestCloudDeleteByQuery, SuggesterTSTTest,
BlockJoinFacetRandomTest, TestLFUCache, DistributedFacetPivotWhiteBoxTest,
PhrasesIdentificationComponentTest, TestPivotHelperCode, ConnectionReuseTest,
HttpPartitionTest, IndexBasedSpellCheckerTest, TestPayloadScoreQParserPlugin,
DirectUpdateHandlerTest, AddBlockUpdateTest, TestJsonFacetsStatsParsing,
NodeMarkersRegistrationTest, NoCacheHeaderTest, TestSafeXMLParsing,
CreateCollectionCleanupTest, TestShardHandlerFactory, TestZkAclsWithHadoopAuth,
BitVectorTest, TestSizeLimitedDistributedMap, TestDistributedGrouping,
HdfsRecoveryZkTest, ClusterStateTest, TestSegmentSorting,
ClassificationUpdateProcessorTest, TestFieldCacheVsDocValues, AddReplicaTest,
RootFieldTest, NodeAddedTriggerIntegrationTest, TestFieldCacheSort,
SecurityConfHandlerTest, TestSolrQueryParser, SimplePostToolTest,
OpenExchangeRatesOrgProviderTest, ConvertedLegacyTest, RegexBoostProcessorTest,
TestSimTriggerIntegration]
[junit4] Completed [387/837 (1!)] on J0 in 2.28s, 11 tests, 1 failure, 9
skipped <<< FAILURES!
[...truncated 9893 lines...]
[javac] Compiling 6 source files to
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/contrib/solr-prometheus-exporter/classes/java
[javac]
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/contrib/prometheus-exporter/src/java/org/apache/solr/prometheus/collector/SolrCollector.java:21:
error: cannot find symbol
[javac] import org.apache.solr.core.Config;
[javac] ^
[javac] symbol: class Config
[javac] location: package org.apache.solr.core
[javac]
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/contrib/prometheus-exporter/src/java/org/apache/solr/prometheus/collector/SolrCollector.java:60:
error: cannot find symbol
[javac] private Config config;
[javac] ^
[javac] symbol: class Config
[javac] location: class SolrCollector
[javac]
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/contrib/prometheus-exporter/src/java/org/apache/solr/prometheus/collector/SolrCollector.java:68:
error: cannot find symbol
[javac] public SolrCollector(SolrClient solrClient, Config config, int
numThreads) {
[javac] ^
[javac] symbol: class Config
[javac] location: class SolrCollector
[javac]
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/contrib/prometheus-exporter/src/java/org/apache/solr/prometheus/exporter/SolrExporter.java:32:
error: cannot find symbol
[javac] import org.apache.solr.core.Config;
[javac] ^
[javac] symbol: class Config
[javac] location: package org.apache.solr.core
[javac]
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/contrib/prometheus-exporter/src/java/org/apache/solr/prometheus/exporter/SolrExporter.java:87:
error: cannot find symbol
[javac] private Config config;
[javac] ^
[javac] symbol: class Config
[javac] location: class SolrExporter
[javac]
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/contrib/prometheus-exporter/src/java/org/apache/solr/prometheus/exporter/SolrExporter.java:118:
error: cannot find symbol
[javac] this.config = new Config(this.loader,
configPath.getFileName().toString());
[javac] ^
[javac] symbol: class Config
[javac] location: class SolrExporter
[javac] Note:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/contrib/prometheus-exporter/src/java/org/apache/solr/prometheus/exporter/SolrExporter.java
uses or overrides a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 6 errors
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/build.xml:633: The following
error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/build.xml:577: The following
error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/build.xml:59: The following
error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build.xml:287: The
following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/common-build.xml:479:
The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/common-build.xml:558:
The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:558:
The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:2056:
Compile failed; see the compiler error output for details.
Total time: 91 minutes 27 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Setting
ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Setting
ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
Setting
ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting
ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting
ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting
ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
