[
https://issues.apache.org/jira/browse/BROOKLYN-272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aled Sage updated BROOKLYN-272:
-------------------------------
Description:
There are several non-deterministic tests that fail occasionally in the apache
jenkins build. We should fix those tests: either the underlying bug if there is
one, or the way to the test has been written to make it more deterministic.
Having build failures unrelated to the changes being made in a PR (and on
master) is extremely disruptive for development. We must avoid that wherever
possible.
I therefore suggest we disable these non-deterministic failing tests
immediately, and use this jira issue to track fixing them. We can annotate the
tests with
{noformat}
@Test(groups={"WIP", "Non-deterministic-failure"})
{noformat}
to make them easy to find. (note the "Non-deterministic-failure" group is not
used by any profiles, it's just there for searching purposes).
The tests I have identified from recent build failures are:
* {{BasicStartableTest.testTransitionsThroughLifecycles}} - see
https://issues.apache.org/jira/browse/BROOKLYN-256
* {{AbstractGeoDnsServiceTest.testFiltersForRunningEntities}}, failing with:
{noformat}
org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: failed
succeeds-eventually, 69 attempts, 30003ms elapsed: AssertionError:
val={R48yHLqg=<address-ignored>, eBHFc4Qd=<address-ignored>}
at org.apache.brooklyn.test.Asserts.fail(Asserts.java:721)
at org.apache.brooklyn.test.Asserts.assertTrue(Asserts.java:703)
at
org.apache.brooklyn.core.entity.EntityAsserts$2.run(EntityAsserts.java:92)
at
org.apache.brooklyn.test.Asserts$RunnableAdapter.call(Asserts.java:1277)
at
org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:930)
at
org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:854)
at
org.apache.brooklyn.core.entity.EntityAsserts.assertAttributeEventually(EntityAsserts.java:89)
at
org.apache.brooklyn.core.entity.EntityAsserts.assertAttributeEventually(EntityAsserts.java:84)
at
org.apache.brooklyn.entity.dns.AbstractGeoDnsServiceTest.testFiltersForRunningEntities(AbstractGeoDnsServiceTest.java:263)
{noformat}
* {{LoadBalancingPolicyConcurrencyTest.testConcurrentlyAddContainers}}, failing
with:
{noformat}
org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: failed
succeeds-eventually, 29 attempts, 10002ms elapsed: AssertionError:
actual=[18.0, 21.0, 19.0, 0.0, 20.0, 0.0, 20.0, 36.0, 19.0, 0.0, 19.0, 21.0,
21.0, 19.0, 21.0, 36.0, 21.0, 19.0, 18.0, 36.0]; expected=[20.0, 20.0, 20.0,
20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0,
20.0, 20.0, 20.0, 20.0] expected [20.0] but found [0.0]
at org.testng.Assert.fail(Assert.java:94)
at org.testng.Assert.failNotEquals(Assert.java:494)
at org.testng.Assert.assertEquals(Assert.java:207)
at
org.apache.brooklyn.policy.loadbalancing.AbstractLoadBalancingPolicyTest.assertWorkrates(AbstractLoadBalancingPolicyTest.java:122)
at
org.apache.brooklyn.policy.loadbalancing.AbstractLoadBalancingPolicyTest$2.run(AbstractLoadBalancingPolicyTest.java:138)
at
org.apache.brooklyn.test.Asserts$RunnableAdapter.call(Asserts.java:1277)
at
org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:930)
at
org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:854)
at
org.apache.brooklyn.policy.loadbalancing.AbstractLoadBalancingPolicyTest.assertWorkratesEventually(AbstractLoadBalancingPolicyTest.java:136)
at
org.apache.brooklyn.policy.loadbalancing.LoadBalancingPolicyConcurrencyTest.testConcurrentlyAddContainers(LoadBalancingPolicyConcurrencyTest.java:101)
{noformat}
* {{PollerTest.testFeedContinuesWhenPollerThrows}}, failing with:
{noformat}
testFeedContinuesWhenPollerThrows(org.apache.brooklyn.core.feed.PollerTest)
Time elapsed: 3.556 sec <<< FAILURE!
org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: failed
succeeds-eventually, 4 attempts, 105ms elapsed: AssertionError:
entity=FeedExceptionEntityImpl{id=dYG0SKbP}; attribute=Sensor: flag
(java.lang.Boolean) expected [false] but found [true]
at org.apache.brooklyn.test.Asserts.fail(Asserts.java:721)
at org.apache.brooklyn.test.Asserts.failNotEquals(Asserts.java:114)
at org.apache.brooklyn.test.Asserts.assertEquals(Asserts.java:436)
at
org.apache.brooklyn.core.entity.EntityAsserts.assertAttributeEquals(EntityAsserts.java:54)
at
org.apache.brooklyn.core.entity.EntityAsserts$1.run(EntityAsserts.java:70)
at
org.apache.brooklyn.test.Asserts$RunnableAdapter.call(Asserts.java:1277)
at org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:930)
at org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:854)
at
org.apache.brooklyn.core.entity.EntityAsserts.assertAttributeEqualsEventually(EntityAsserts.java:67)
at
org.apache.brooklyn.core.feed.PollerTest.testFeedContinuesWhenPollerThrows(PollerTest.java:65)
{noformat}
was:
There are several non-deterministic tests that fail occasionally in the apache
jenkins build. We should fix those tests: either the underlying bug if there is
one, or the way to the test has been written to make it more deterministic.
Having build failures unrelated to the changes being made in a PR (and on
master) is extremely disruptive for development. We must avoid that wherever
possible.
I therefore suggest we disable these non-deterministic failing tests
immediately, and use this jira issue to track fixing them. We can annotate the
tests with {{@Test(groups={"WIP", "Non-deterministic-failure"})}} to make them
easy to find. (note the "Non-deterministic-failure" group is not used by any
profiles, it's just there for searching purposes).
The tests I have identified from recent build failures are:
* {{BasicStartableTest.testTransitionsThroughLifecycles}} - see
https://issues.apache.org/jira/browse/BROOKLYN-256
* {{AbstractGeoDnsServiceTest.testFiltersForRunningEntities}}, failing with:
{noformat}
org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: failed
succeeds-eventually, 69 attempts, 30003ms elapsed: AssertionError:
val={R48yHLqg=<address-ignored>, eBHFc4Qd=<address-ignored>}
at org.apache.brooklyn.test.Asserts.fail(Asserts.java:721)
at org.apache.brooklyn.test.Asserts.assertTrue(Asserts.java:703)
at
org.apache.brooklyn.core.entity.EntityAsserts$2.run(EntityAsserts.java:92)
at
org.apache.brooklyn.test.Asserts$RunnableAdapter.call(Asserts.java:1277)
at
org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:930)
at
org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:854)
at
org.apache.brooklyn.core.entity.EntityAsserts.assertAttributeEventually(EntityAsserts.java:89)
at
org.apache.brooklyn.core.entity.EntityAsserts.assertAttributeEventually(EntityAsserts.java:84)
at
org.apache.brooklyn.entity.dns.AbstractGeoDnsServiceTest.testFiltersForRunningEntities(AbstractGeoDnsServiceTest.java:263)
{noformat}
* {{LoadBalancingPolicyConcurrencyTest.testConcurrentlyAddContainers}}, failing
with:
{noformat}
org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: failed
succeeds-eventually, 29 attempts, 10002ms elapsed: AssertionError:
actual=[18.0, 21.0, 19.0, 0.0, 20.0, 0.0, 20.0, 36.0, 19.0, 0.0, 19.0, 21.0,
21.0, 19.0, 21.0, 36.0, 21.0, 19.0, 18.0, 36.0]; expected=[20.0, 20.0, 20.0,
20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0,
20.0, 20.0, 20.0, 20.0] expected [20.0] but found [0.0]
at org.testng.Assert.fail(Assert.java:94)
at org.testng.Assert.failNotEquals(Assert.java:494)
at org.testng.Assert.assertEquals(Assert.java:207)
at
org.apache.brooklyn.policy.loadbalancing.AbstractLoadBalancingPolicyTest.assertWorkrates(AbstractLoadBalancingPolicyTest.java:122)
at
org.apache.brooklyn.policy.loadbalancing.AbstractLoadBalancingPolicyTest$2.run(AbstractLoadBalancingPolicyTest.java:138)
at
org.apache.brooklyn.test.Asserts$RunnableAdapter.call(Asserts.java:1277)
at
org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:930)
at
org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:854)
at
org.apache.brooklyn.policy.loadbalancing.AbstractLoadBalancingPolicyTest.assertWorkratesEventually(AbstractLoadBalancingPolicyTest.java:136)
at
org.apache.brooklyn.policy.loadbalancing.LoadBalancingPolicyConcurrencyTest.testConcurrentlyAddContainers(LoadBalancingPolicyConcurrencyTest.java:101)
{noformat}
* {{PollerTest.testFeedContinuesWhenPollerThrows}}, failing with:
{noformat}
testFeedContinuesWhenPollerThrows(org.apache.brooklyn.core.feed.PollerTest)
Time elapsed: 3.556 sec <<< FAILURE!
org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: failed
succeeds-eventually, 4 attempts, 105ms elapsed: AssertionError:
entity=FeedExceptionEntityImpl{id=dYG0SKbP}; attribute=Sensor: flag
(java.lang.Boolean) expected [false] but found [true]
at org.apache.brooklyn.test.Asserts.fail(Asserts.java:721)
at org.apache.brooklyn.test.Asserts.failNotEquals(Asserts.java:114)
at org.apache.brooklyn.test.Asserts.assertEquals(Asserts.java:436)
at
org.apache.brooklyn.core.entity.EntityAsserts.assertAttributeEquals(EntityAsserts.java:54)
at
org.apache.brooklyn.core.entity.EntityAsserts$1.run(EntityAsserts.java:70)
at
org.apache.brooklyn.test.Asserts$RunnableAdapter.call(Asserts.java:1277)
at org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:930)
at org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:854)
at
org.apache.brooklyn.core.entity.EntityAsserts.assertAttributeEqualsEventually(EntityAsserts.java:67)
at
org.apache.brooklyn.core.feed.PollerTest.testFeedContinuesWhenPollerThrows(PollerTest.java:65)
{noformat}
> Non-deterministic test failures in jenkins
> ------------------------------------------
>
> Key: BROOKLYN-272
> URL: https://issues.apache.org/jira/browse/BROOKLYN-272
> Project: Brooklyn
> Issue Type: Bug
> Reporter: Aled Sage
>
> There are several non-deterministic tests that fail occasionally in the
> apache jenkins build. We should fix those tests: either the underlying bug if
> there is one, or the way to the test has been written to make it more
> deterministic.
> Having build failures unrelated to the changes being made in a PR (and on
> master) is extremely disruptive for development. We must avoid that wherever
> possible.
> I therefore suggest we disable these non-deterministic failing tests
> immediately, and use this jira issue to track fixing them. We can annotate
> the tests with
> {noformat}
> @Test(groups={"WIP", "Non-deterministic-failure"})
> {noformat}
> to make them easy to find. (note the "Non-deterministic-failure" group is
> not used by any profiles, it's just there for searching purposes).
> The tests I have identified from recent build failures are:
> * {{BasicStartableTest.testTransitionsThroughLifecycles}} - see
> https://issues.apache.org/jira/browse/BROOKLYN-256
> * {{AbstractGeoDnsServiceTest.testFiltersForRunningEntities}}, failing with:
> {noformat}
> org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: failed
> succeeds-eventually, 69 attempts, 30003ms elapsed: AssertionError:
> val={R48yHLqg=<address-ignored>, eBHFc4Qd=<address-ignored>}
> at org.apache.brooklyn.test.Asserts.fail(Asserts.java:721)
> at org.apache.brooklyn.test.Asserts.assertTrue(Asserts.java:703)
> at
> org.apache.brooklyn.core.entity.EntityAsserts$2.run(EntityAsserts.java:92)
> at
> org.apache.brooklyn.test.Asserts$RunnableAdapter.call(Asserts.java:1277)
> at
> org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:930)
> at
> org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:854)
> at
> org.apache.brooklyn.core.entity.EntityAsserts.assertAttributeEventually(EntityAsserts.java:89)
> at
> org.apache.brooklyn.core.entity.EntityAsserts.assertAttributeEventually(EntityAsserts.java:84)
> at
> org.apache.brooklyn.entity.dns.AbstractGeoDnsServiceTest.testFiltersForRunningEntities(AbstractGeoDnsServiceTest.java:263)
> {noformat}
> * {{LoadBalancingPolicyConcurrencyTest.testConcurrentlyAddContainers}},
> failing with:
> {noformat}
> org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: failed
> succeeds-eventually, 29 attempts, 10002ms elapsed: AssertionError:
> actual=[18.0, 21.0, 19.0, 0.0, 20.0, 0.0, 20.0, 36.0, 19.0, 0.0, 19.0, 21.0,
> 21.0, 19.0, 21.0, 36.0, 21.0, 19.0, 18.0, 36.0]; expected=[20.0, 20.0, 20.0,
> 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0,
> 20.0, 20.0, 20.0, 20.0] expected [20.0] but found [0.0]
> at org.testng.Assert.fail(Assert.java:94)
> at org.testng.Assert.failNotEquals(Assert.java:494)
> at org.testng.Assert.assertEquals(Assert.java:207)
> at
> org.apache.brooklyn.policy.loadbalancing.AbstractLoadBalancingPolicyTest.assertWorkrates(AbstractLoadBalancingPolicyTest.java:122)
> at
> org.apache.brooklyn.policy.loadbalancing.AbstractLoadBalancingPolicyTest$2.run(AbstractLoadBalancingPolicyTest.java:138)
> at
> org.apache.brooklyn.test.Asserts$RunnableAdapter.call(Asserts.java:1277)
> at
> org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:930)
> at
> org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:854)
> at
> org.apache.brooklyn.policy.loadbalancing.AbstractLoadBalancingPolicyTest.assertWorkratesEventually(AbstractLoadBalancingPolicyTest.java:136)
> at
> org.apache.brooklyn.policy.loadbalancing.LoadBalancingPolicyConcurrencyTest.testConcurrentlyAddContainers(LoadBalancingPolicyConcurrencyTest.java:101)
> {noformat}
> * {{PollerTest.testFeedContinuesWhenPollerThrows}}, failing with:
> {noformat}
>
> testFeedContinuesWhenPollerThrows(org.apache.brooklyn.core.feed.PollerTest)
> Time elapsed: 3.556 sec <<< FAILURE!
> org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: failed
> succeeds-eventually, 4 attempts, 105ms elapsed: AssertionError:
> entity=FeedExceptionEntityImpl{id=dYG0SKbP}; attribute=Sensor: flag
> (java.lang.Boolean) expected [false] but found [true]
> at org.apache.brooklyn.test.Asserts.fail(Asserts.java:721)
> at org.apache.brooklyn.test.Asserts.failNotEquals(Asserts.java:114)
> at org.apache.brooklyn.test.Asserts.assertEquals(Asserts.java:436)
> at
> org.apache.brooklyn.core.entity.EntityAsserts.assertAttributeEquals(EntityAsserts.java:54)
> at
> org.apache.brooklyn.core.entity.EntityAsserts$1.run(EntityAsserts.java:70)
> at
> org.apache.brooklyn.test.Asserts$RunnableAdapter.call(Asserts.java:1277)
> at
> org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:930)
> at
> org.apache.brooklyn.test.Asserts.succeedsEventually(Asserts.java:854)
> at
> org.apache.brooklyn.core.entity.EntityAsserts.assertAttributeEqualsEventually(EntityAsserts.java:67)
> at
> org.apache.brooklyn.core.feed.PollerTest.testFeedContinuesWhenPollerThrows(PollerTest.java:65)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
