Aled Sage created BROOKLYN-564:
----------------------------------
Summary:
ElectPrimaryTest.testSelectionModeFailoverReelectWithPreference fails
non-deterministically
Key: BROOKLYN-564
URL: https://issues.apache.org/jira/browse/BROOKLYN-564
Project: Brooklyn
Issue Type: Bug
Reporter: Aled Sage
Priority: Minor
In Brooklyn 1.0.0-SNAPSHOT,
{{ElectPrimaryTest.testSelectionModeFailoverReelectWithPreference fails
non-deterministically}} sometimes fails in jenkins:
{noformat}
2017-11-13 11:45:06,214 INFO TESTNG INVOKING CONFIGURATION: "Surefire test" -
@BeforeMethod org.apache.brooklyn.camp.brooklyn.AbstractYamlRebindTest.setUp()
2017-11-13 11:45:06,214 INFO Added external config supplier named
'brooklyn-demo-sample':
org.apache.brooklyn.core.config.external.InPlaceExternalConfigSupplier@380e4e8a
2017-11-13 11:45:06,219 INFO Rebind check: no existing state; will persist new
items to /tmp/ElectPrimaryTest-UgLc
2017-11-13 11:45:06,220 INFO No Camp-YAML parser registered for parsing
catalog item DSL; skipping DSL-parsing
2017-11-13 11:45:06,226 INFO Test class
org.apache.brooklyn.camp.brooklyn.test.policy.failover.ElectPrimaryTest
persisting to /tmp/ElectPrimaryTest-UgLc
2017-11-13 11:45:06,227 INFO TESTNG PASSED CONFIGURATION: "Surefire test" -
@BeforeMethod org.apache.brooklyn.camp.brooklyn.AbstractYamlRebindTest.setUp()
finished in 13 ms
2017-11-13 11:45:06,227 INFO TESTNG INVOKING: "Surefire test" -
org.apache.brooklyn.camp.brooklyn.test.policy.failover.ElectPrimaryTest.testSelectionModeFailoverReelectWithPreference()
2017-11-13 11:45:06,279 INFO Test created app, and will now start
BasicApplicationImpl{id=nq3a3nshq1}
2017-11-13 11:45:06,307 INFO Detected new primary
TestEntityImpl{id=rg0witaqno} at BasicApplicationImpl{id=nq3a3nshq1}
(previously had null)
2017-11-13 11:45:06,315 INFO Primary TestEntityImpl{id=rg0witaqno} at
BasicApplicationImpl{id=nq3a3nshq1} detected as healthy
2017-11-13 11:45:06,329 INFO Started application
BasicApplicationImpl{id=nq3a3nshq1}
2017-11-13 11:45:06,330 INFO Detected new primary
TestEntityImpl{id=cijb9r79z1} at BasicApplicationImpl{id=nq3a3nshq1}
(previously had TestEntityImpl{id=rg0witaqno})
2017-11-13 11:45:06,331 INFO Detected new primary
TestEntityImpl{id=rg0witaqno} at BasicApplicationImpl{id=nq3a3nshq1}
(previously had TestEntityImpl{id=cijb9r79z1})
2017-11-13 11:45:36,343 INFO succeedsEventually exceeded max attempts or
timeout - 75 attempts lasting 30000 ms, for
RunnableAdapter(org.apache.brooklyn.core.entity.EntityAsserts$2@2eae3041)
2017-11-13 11:45:36,344 INFO failed succeeds-eventually, 75 attempts, 30000ms
elapsed (rethrowing): java.lang.AssertionError: attribute=Sensor: primary
(org.apache.brooklyn.api.entity.Entity); val=TestEntityImpl{id=rg0witaqno}
2017-11-13 11:45:36,346 ERROR Failed: java.lang.AssertionError: failed
succeeds-eventually, 75 attempts, 30001ms elapsed: AssertionError:
attribute=Sensor: primary (org.apache.brooklyn.api.entity.Entity);
val=TestEntityImpl{id=rg0witaqno}
java.lang.AssertionError: failed succeeds-eventually, 75 attempts, 30001ms
elapsed: AssertionError: attribute=Sensor: primary
(org.apache.brooklyn.api.entity.Entity); val=TestEntityImpl{id=rg0witaqno}
at
org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:1009)
~[brooklyn-utils-common-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
Caused by: java.lang.AssertionError: attribute=Sensor: primary
(org.apache.brooklyn.api.entity.Entity); val=TestEntityImpl{id=rg0witaqno}
at org.apache.brooklyn.test.Asserts.fail(Asserts.java:755)
~[brooklyn-utils-common-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
BasicApplicationImpl{id=nq3a3nshq1} nq3a3nshq1
displayName = Application (nq3a3nshq1)
locations = []
brooklyn.wrapper_app = true
camp.template.id = WpI0y0hC
quorum.running = QuorumCheck[require=1,0.0%]
quorum.up = QuorumCheck[require=1,0.0%]
application.id: nq3a3nshq1
entity.id: nq3a3nshq1
primary: TestEntityImpl{id=rg0witaqno}
service.isUp: true
service.state: running
service.state.expected: running @ 1510573506343 / Mon Nov 13 11:45:06 UTC
2017
Policies:
ElectPrimaryPolicy{id=gcebjkm4dq, name=null, running=true}
primary.selection.mode[ConfigKey:org.apache.brooklyn.policy.failover.ElectPrimaryConfig$SelectionMode]
= FAILOVER
primary.stopped.wait.timeout[ConfigKey:org.apache.brooklyn.util.time.Duration]
= 0ms
Enrichers:
Transformer{uniqueTag=service.isUp if no service.notUp.indicators,
running=true, entity=BasicApplicationImpl{id=nq3a3nshq1}, id=iq1mvcb263}
enricher.sourceSensor[ConfigKey:org.apache.brooklyn.api.sensor.Sensor] =
Sensor: service.notUp.indicators (java.util.Map)
enricher.suppressDuplicates[ConfigKey:java.lang.Boolean] = true
enricher.targetSensor[ConfigKey:org.apache.brooklyn.api.sensor.Sensor] =
Sensor: service.isUp (java.lang.Boolean)
enricher.transformation[ConfigKey:com.google.common.base.Function] =
if[{Predicates.not(Predicates.isNull())=forPredicate(Predicates.equalTo(0)(sizeFunction(keys)))}]-else[UNCHANGED]
ComputeServiceState{uniqueTag=service.state.actual, running=true,
entity=BasicApplicationImpl{id=nq3a3nshq1}, id=d90sql52vz}
ComputeServiceIndicatorsFromChildrenAndMembers{uniqueTag=service-lifecycle-indicators-from-children-and-members,
running=true, entity=BasicApplicationImpl{id=nq3a3nshq1}, id=g6pgaxo97d}
enricher.aggregating.fromChildren[ConfigKey:java.lang.Boolean] = true
enricher.aggregating.fromMembers[ConfigKey:java.lang.Boolean] = true
enricher.service_state.children_and_members.quorum.running[ConfigKey:org.apache.brooklyn.util.collections.QuorumCheck]
= QuorumCheck[all;require=0,100.0%]
enricher.service_state.children_and_members.quorum.up[ConfigKey:org.apache.brooklyn.util.collections.QuorumCheck]
= QuorumCheck[all;require=0,100.0%]
PrimaryRunningEnricher{running=true,
entity=BasicApplicationImpl{id=nq3a3nshq1}, id=wcpg5ordwa}
TestEntityImpl{id=rg0witaqno} rg0witaqno
displayName = a
catalogItemId = item-from-test:0.0.0-SNAPSHOT
locations = []
camp.template.id = pfqqeadI
ha.primary.weight = 1
application.id: nq3a3nshq1
catalog.id: item-from-test:0.0.0-SNAPSHOT
entity.id: rg0witaqno
service.isUp: true
service.state: running
service.state.expected: running @ 1510573506297 / Mon Nov 13 11:45:06 UTC
2017
test.attributeAndConfString: defaultval
Enrichers:
Transformer{uniqueTag=service.isUp if no service.notUp.indicators,
running=true, entity=TestEntityImpl{id=rg0witaqno}, id=bjbxe2ddrz}
enricher.sourceSensor[ConfigKey:org.apache.brooklyn.api.sensor.Sensor] =
Sensor: service.notUp.indicators (java.util.Map)
enricher.suppressDuplicates[ConfigKey:java.lang.Boolean] = true
enricher.targetSensor[ConfigKey:org.apache.brooklyn.api.sensor.Sensor] =
Sensor: service.isUp (java.lang.Boolean)
enricher.transformation[ConfigKey:com.google.common.base.Function]
=
if[{Predicates.not(Predicates.isNull())=forPredicate(Predicates.equalTo(0)(sizeFunction(keys)))}]-else[UNCHANGED]
ComputeServiceState{uniqueTag=service.state.actual, running=true,
entity=TestEntityImpl{id=rg0witaqno}, id=qdmiyq1sp8}
TestEntityImpl{id=cijb9r79z1} cijb9r79z1
displayName = b
catalogItemId = item-from-test:0.0.0-SNAPSHOT
locations = []
camp.template.id = sjvkU3bT
application.id: nq3a3nshq1
catalog.id: item-from-test:0.0.0-SNAPSHOT
entity.id: cijb9r79z1
ha.primary.weight: 2.0
service.isUp: true
service.state: running
service.state.expected: running @ 1510573506302 / Mon Nov 13 11:45:06 UTC
2017
test.attributeAndConfString: defaultval
Enrichers:
Transformer{uniqueTag=service.isUp if no service.notUp.indicators,
running=true, entity=TestEntityImpl{id=cijb9r79z1}, id=h1z8b6pteu}
enricher.sourceSensor[ConfigKey:org.apache.brooklyn.api.sensor.Sensor] =
Sensor: service.notUp.indicators (java.util.Map)
enricher.suppressDuplicates[ConfigKey:java.lang.Boolean] = true
enricher.targetSensor[ConfigKey:org.apache.brooklyn.api.sensor.Sensor] =
Sensor: service.isUp (java.lang.Boolean)
enricher.transformation[ConfigKey:com.google.common.base.Function]
=
if[{Predicates.not(Predicates.isNull())=forPredicate(Predicates.equalTo(0)(sizeFunction(keys)))}]-else[UNCHANGED]
ComputeServiceState{uniqueTag=service.state.actual, running=true,
entity=TestEntityImpl{id=cijb9r79z1}, id=n27syuicif}
Standard Error
java.lang.AssertionError: failed succeeds-eventually, 75 attempts, 30001ms
elapsed: AssertionError: attribute=Sensor: primary
(org.apache.brooklyn.api.entity.Entity); val=TestEntityImpl{id=rg0witaqno}
at
org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:1009)
at org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:892)
at
org.apache.brooklyn.core.entity.EntityAsserts.assertAttributeEventually(EntityAsserts.java:93)
at
org.apache.brooklyn.core.entity.EntityAsserts.assertAttributeEventually(EntityAsserts.java:88)
at
org.apache.brooklyn.camp.brooklyn.test.policy.failover.ElectPrimaryTest.runSelectionModeTest(ElectPrimaryTest.java:334)
at
org.apache.brooklyn.camp.brooklyn.test.policy.failover.ElectPrimaryTest.testSelectionModeFailoverReelectWithPreference(ElectPrimaryTest.java:292)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:645)
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:851)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1177)
at
org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
at org.testng.TestRunner.privateRun(TestRunner.java:756)
at org.testng.TestRunner.run(TestRunner.java:610)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:387)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:382)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:340)
at org.testng.SuiteRunner.run(SuiteRunner.java:289)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1293)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1218)
at org.testng.TestNG.runSuites(TestNG.java:1133)
at org.testng.TestNG.run(TestNG.java:1104)
at
org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:132)
at
org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeMulti(TestNGDirectoryTestSuite.java:193)
at
org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:94)
at
org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:147)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121)
Caused by: java.lang.AssertionError: attribute=Sensor: primary
(org.apache.brooklyn.api.entity.Entity); val=TestEntityImpl{id=rg0witaqno}
at org.apache.brooklyn.test.Asserts.fail(Asserts.java:755)
at org.apache.brooklyn.test.Asserts.assertTrue(Asserts.java:734)
at
org.apache.brooklyn.core.entity.EntityAsserts.assertAttribute(EntityAsserts.java:103)
at
org.apache.brooklyn.core.entity.EntityAsserts$2.run(EntityAsserts.java:95)
at
org.apache.brooklyn.test.Asserts$RunnableAdapter.call(Asserts.java:1363)
at org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:970)
... 34 more
{noformat}
The underlying reason is that the {{ElectPrimaryEffector}} is executed multiple
times concurrently (it could be up to three: first from
{{ElectPrimaryPolicy.setEntity}} calling {{rescanRequest}}, next from a change
of a child's sensor, and third in the {{ElectPrimaryTest.runSelectionModeTest}}
when it explicitly calls the effector with mode=best; in this case I think it's
just the last two; the first doesn't happen concurrently because the policy is
added before the entity is managed).
When the effectors execute concurrently, the mode=best invocation can set the
primary first, and then the one triggered by sensor-changed (which had already
decided who the primary should be) sees that the current-active does not match
what it thinks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
