[
https://issues.apache.org/jira/browse/BROOKLYN-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16251298#comment-16251298
]
Aled Sage commented on BROOKLYN-564:
------------------------------------
This is mainly just an issue with how the test is written, but could be an
issue for real users if they invoke the effector manually. The particularly bad
news if invoking it manually is that there is no mutex to stop the policy
trying to do something at the same time, so you can end up with promote/demote
effectors being called concurrently, which could lead to really bad behaviour!
If it's just the policy executing stuff, then it should be ok because the
policy is conceptually single-threaded (except where we invoke it explicitly in
setEntity, but if the policy exists from the beginning then it should happen
before the entity is managed and thus before any subscriptions kick-in).
> ElectPrimaryTest.testSelectionModeFailoverReelectWithPreference fails
> non-deterministically
> -------------------------------------------------------------------------------------------
>
> Key: BROOKLYN-564
> URL: https://issues.apache.org/jira/browse/BROOKLYN-564
> Project: Brooklyn
> Issue Type: Bug
> Reporter: Aled Sage
> Priority: Minor
>
> In Brooklyn 1.0.0-SNAPSHOT,
> {{ElectPrimaryTest.testSelectionModeFailoverReelectWithPreference fails
> non-deterministically}} sometimes fails in jenkins:
> {noformat}
> 2017-11-13 11:45:06,214 INFO TESTNG INVOKING CONFIGURATION: "Surefire test"
> - @BeforeMethod
> org.apache.brooklyn.camp.brooklyn.AbstractYamlRebindTest.setUp()
> 2017-11-13 11:45:06,214 INFO Added external config supplier named
> 'brooklyn-demo-sample':
> org.apache.brooklyn.core.config.external.InPlaceExternalConfigSupplier@380e4e8a
> 2017-11-13 11:45:06,219 INFO Rebind check: no existing state; will persist
> new items to /tmp/ElectPrimaryTest-UgLc
> 2017-11-13 11:45:06,220 INFO No Camp-YAML parser registered for parsing
> catalog item DSL; skipping DSL-parsing
> 2017-11-13 11:45:06,226 INFO Test class
> org.apache.brooklyn.camp.brooklyn.test.policy.failover.ElectPrimaryTest
> persisting to /tmp/ElectPrimaryTest-UgLc
> 2017-11-13 11:45:06,227 INFO TESTNG PASSED CONFIGURATION: "Surefire test" -
> @BeforeMethod
> org.apache.brooklyn.camp.brooklyn.AbstractYamlRebindTest.setUp() finished in
> 13 ms
> 2017-11-13 11:45:06,227 INFO TESTNG INVOKING: "Surefire test" -
> org.apache.brooklyn.camp.brooklyn.test.policy.failover.ElectPrimaryTest.testSelectionModeFailoverReelectWithPreference()
> 2017-11-13 11:45:06,279 INFO Test created app, and will now start
> BasicApplicationImpl{id=nq3a3nshq1}
> 2017-11-13 11:45:06,307 INFO Detected new primary
> TestEntityImpl{id=rg0witaqno} at BasicApplicationImpl{id=nq3a3nshq1}
> (previously had null)
> 2017-11-13 11:45:06,315 INFO Primary TestEntityImpl{id=rg0witaqno} at
> BasicApplicationImpl{id=nq3a3nshq1} detected as healthy
> 2017-11-13 11:45:06,329 INFO Started application
> BasicApplicationImpl{id=nq3a3nshq1}
> 2017-11-13 11:45:06,330 INFO Detected new primary
> TestEntityImpl{id=cijb9r79z1} at BasicApplicationImpl{id=nq3a3nshq1}
> (previously had TestEntityImpl{id=rg0witaqno})
> 2017-11-13 11:45:06,331 INFO Detected new primary
> TestEntityImpl{id=rg0witaqno} at BasicApplicationImpl{id=nq3a3nshq1}
> (previously had TestEntityImpl{id=cijb9r79z1})
> 2017-11-13 11:45:36,343 INFO succeedsEventually exceeded max attempts or
> timeout - 75 attempts lasting 30000 ms, for
> RunnableAdapter(org.apache.brooklyn.core.entity.EntityAsserts$2@2eae3041)
> 2017-11-13 11:45:36,344 INFO failed succeeds-eventually, 75 attempts,
> 30000ms elapsed (rethrowing): java.lang.AssertionError: attribute=Sensor:
> primary (org.apache.brooklyn.api.entity.Entity);
> val=TestEntityImpl{id=rg0witaqno}
> 2017-11-13 11:45:36,346 ERROR Failed: java.lang.AssertionError: failed
> succeeds-eventually, 75 attempts, 30001ms elapsed: AssertionError:
> attribute=Sensor: primary (org.apache.brooklyn.api.entity.Entity);
> val=TestEntityImpl{id=rg0witaqno}
> java.lang.AssertionError: failed succeeds-eventually, 75 attempts, 30001ms
> elapsed: AssertionError: attribute=Sensor: primary
> (org.apache.brooklyn.api.entity.Entity); val=TestEntityImpl{id=rg0witaqno}
> at
> org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:1009)
> ~[brooklyn-utils-common-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> Caused by: java.lang.AssertionError: attribute=Sensor: primary
> (org.apache.brooklyn.api.entity.Entity); val=TestEntityImpl{id=rg0witaqno}
> at org.apache.brooklyn.test.Asserts.fail(Asserts.java:755)
> ~[brooklyn-utils-common-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> BasicApplicationImpl{id=nq3a3nshq1} nq3a3nshq1
> displayName = Application (nq3a3nshq1)
> locations = []
> brooklyn.wrapper_app = true
> camp.template.id = WpI0y0hC
> quorum.running = QuorumCheck[require=1,0.0%]
> quorum.up = QuorumCheck[require=1,0.0%]
> application.id: nq3a3nshq1
> entity.id: nq3a3nshq1
> primary: TestEntityImpl{id=rg0witaqno}
> service.isUp: true
> service.state: running
> service.state.expected: running @ 1510573506343 / Mon Nov 13 11:45:06 UTC
> 2017
> Policies:
> ElectPrimaryPolicy{id=gcebjkm4dq, name=null, running=true}
>
> primary.selection.mode[ConfigKey:org.apache.brooklyn.policy.failover.ElectPrimaryConfig$SelectionMode]
> = FAILOVER
>
> primary.stopped.wait.timeout[ConfigKey:org.apache.brooklyn.util.time.Duration]
> = 0ms
> Enrichers:
> Transformer{uniqueTag=service.isUp if no service.notUp.indicators,
> running=true, entity=BasicApplicationImpl{id=nq3a3nshq1}, id=iq1mvcb263}
>
> enricher.sourceSensor[ConfigKey:org.apache.brooklyn.api.sensor.Sensor] =
> Sensor: service.notUp.indicators (java.util.Map)
> enricher.suppressDuplicates[ConfigKey:java.lang.Boolean] = true
>
> enricher.targetSensor[ConfigKey:org.apache.brooklyn.api.sensor.Sensor] =
> Sensor: service.isUp (java.lang.Boolean)
> enricher.transformation[ConfigKey:com.google.common.base.Function]
> =
> if[{Predicates.not(Predicates.isNull())=forPredicate(Predicates.equalTo(0)(sizeFunction(keys)))}]-else[UNCHANGED]
> ComputeServiceState{uniqueTag=service.state.actual, running=true,
> entity=BasicApplicationImpl{id=nq3a3nshq1}, id=d90sql52vz}
>
> ComputeServiceIndicatorsFromChildrenAndMembers{uniqueTag=service-lifecycle-indicators-from-children-and-members,
> running=true, entity=BasicApplicationImpl{id=nq3a3nshq1}, id=g6pgaxo97d}
> enricher.aggregating.fromChildren[ConfigKey:java.lang.Boolean] =
> true
> enricher.aggregating.fromMembers[ConfigKey:java.lang.Boolean] = true
>
> enricher.service_state.children_and_members.quorum.running[ConfigKey:org.apache.brooklyn.util.collections.QuorumCheck]
> = QuorumCheck[all;require=0,100.0%]
>
> enricher.service_state.children_and_members.quorum.up[ConfigKey:org.apache.brooklyn.util.collections.QuorumCheck]
> = QuorumCheck[all;require=0,100.0%]
> PrimaryRunningEnricher{running=true,
> entity=BasicApplicationImpl{id=nq3a3nshq1}, id=wcpg5ordwa}
> TestEntityImpl{id=rg0witaqno} rg0witaqno
> displayName = a
> catalogItemId = item-from-test:0.0.0-SNAPSHOT
> locations = []
> camp.template.id = pfqqeadI
> ha.primary.weight = 1
> application.id: nq3a3nshq1
> catalog.id: item-from-test:0.0.0-SNAPSHOT
> entity.id: rg0witaqno
> service.isUp: true
> service.state: running
> service.state.expected: running @ 1510573506297 / Mon Nov 13 11:45:06
> UTC 2017
> test.attributeAndConfString: defaultval
> Enrichers:
> Transformer{uniqueTag=service.isUp if no service.notUp.indicators,
> running=true, entity=TestEntityImpl{id=rg0witaqno}, id=bjbxe2ddrz}
>
> enricher.sourceSensor[ConfigKey:org.apache.brooklyn.api.sensor.Sensor] =
> Sensor: service.notUp.indicators (java.util.Map)
> enricher.suppressDuplicates[ConfigKey:java.lang.Boolean] = true
>
> enricher.targetSensor[ConfigKey:org.apache.brooklyn.api.sensor.Sensor] =
> Sensor: service.isUp (java.lang.Boolean)
>
> enricher.transformation[ConfigKey:com.google.common.base.Function] =
> if[{Predicates.not(Predicates.isNull())=forPredicate(Predicates.equalTo(0)(sizeFunction(keys)))}]-else[UNCHANGED]
> ComputeServiceState{uniqueTag=service.state.actual, running=true,
> entity=TestEntityImpl{id=rg0witaqno}, id=qdmiyq1sp8}
> TestEntityImpl{id=cijb9r79z1} cijb9r79z1
> displayName = b
> catalogItemId = item-from-test:0.0.0-SNAPSHOT
> locations = []
> camp.template.id = sjvkU3bT
> application.id: nq3a3nshq1
> catalog.id: item-from-test:0.0.0-SNAPSHOT
> entity.id: cijb9r79z1
> ha.primary.weight: 2.0
> service.isUp: true
> service.state: running
> service.state.expected: running @ 1510573506302 / Mon Nov 13 11:45:06
> UTC 2017
> test.attributeAndConfString: defaultval
> Enrichers:
> Transformer{uniqueTag=service.isUp if no service.notUp.indicators,
> running=true, entity=TestEntityImpl{id=cijb9r79z1}, id=h1z8b6pteu}
>
> enricher.sourceSensor[ConfigKey:org.apache.brooklyn.api.sensor.Sensor] =
> Sensor: service.notUp.indicators (java.util.Map)
> enricher.suppressDuplicates[ConfigKey:java.lang.Boolean] = true
>
> enricher.targetSensor[ConfigKey:org.apache.brooklyn.api.sensor.Sensor] =
> Sensor: service.isUp (java.lang.Boolean)
>
> enricher.transformation[ConfigKey:com.google.common.base.Function] =
> if[{Predicates.not(Predicates.isNull())=forPredicate(Predicates.equalTo(0)(sizeFunction(keys)))}]-else[UNCHANGED]
> ComputeServiceState{uniqueTag=service.state.actual, running=true,
> entity=TestEntityImpl{id=cijb9r79z1}, id=n27syuicif}
> Standard Error
> java.lang.AssertionError: failed succeeds-eventually, 75 attempts, 30001ms
> elapsed: AssertionError: attribute=Sensor: primary
> (org.apache.brooklyn.api.entity.Entity); val=TestEntityImpl{id=rg0witaqno}
> at
> org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:1009)
> at org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:892)
> at
> org.apache.brooklyn.core.entity.EntityAsserts.assertAttributeEventually(EntityAsserts.java:93)
> at
> org.apache.brooklyn.core.entity.EntityAsserts.assertAttributeEventually(EntityAsserts.java:88)
> at
> org.apache.brooklyn.camp.brooklyn.test.policy.failover.ElectPrimaryTest.runSelectionModeTest(ElectPrimaryTest.java:334)
> at
> org.apache.brooklyn.camp.brooklyn.test.policy.failover.ElectPrimaryTest.testSelectionModeFailoverReelectWithPreference(ElectPrimaryTest.java:292)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:645)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:851)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1177)
> at
> org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
> at org.testng.TestRunner.privateRun(TestRunner.java:756)
> at org.testng.TestRunner.run(TestRunner.java:610)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:387)
> at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:382)
> at org.testng.SuiteRunner.privateRun(SuiteRunner.java:340)
> at org.testng.SuiteRunner.run(SuiteRunner.java:289)
> at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
> at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
> at org.testng.TestNG.runSuitesSequentially(TestNG.java:1293)
> at org.testng.TestNG.runSuitesLocally(TestNG.java:1218)
> at org.testng.TestNG.runSuites(TestNG.java:1133)
> at org.testng.TestNG.run(TestNG.java:1104)
> at
> org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:132)
> at
> org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeMulti(TestNGDirectoryTestSuite.java:193)
> at
> org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:94)
> at
> org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:147)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121)
> Caused by: java.lang.AssertionError: attribute=Sensor: primary
> (org.apache.brooklyn.api.entity.Entity); val=TestEntityImpl{id=rg0witaqno}
> at org.apache.brooklyn.test.Asserts.fail(Asserts.java:755)
> at org.apache.brooklyn.test.Asserts.assertTrue(Asserts.java:734)
> at
> org.apache.brooklyn.core.entity.EntityAsserts.assertAttribute(EntityAsserts.java:103)
> at
> org.apache.brooklyn.core.entity.EntityAsserts$2.run(EntityAsserts.java:95)
> at
> org.apache.brooklyn.test.Asserts$RunnableAdapter.call(Asserts.java:1363)
> at org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:970)
> ... 34 more
> {noformat}
> The underlying reason is that the {{ElectPrimaryEffector}} is executed
> multiple times concurrently (it could be up to three: first from
> {{ElectPrimaryPolicy.setEntity}} calling {{rescanRequest}}, next from a
> change of a child's sensor, and third in the
> {{ElectPrimaryTest.runSelectionModeTest}} when it explicitly calls the
> effector with mode=best; in this case I think it's just the last two; the
> first doesn't happen concurrently because the policy is added before the
> entity is managed).
> When the effectors execute concurrently, the mode=best invocation can set the
> primary first, and then the one triggered by sensor-changed (which had
> already decided who the primary should be) sees that the current-active does
> not match what it thinks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
