[
https://issues.apache.org/jira/browse/CASSANDRA-12277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15392042#comment-15392042
]
Joshua McKenzie commented on CASSANDRA-12277:
---------------------------------------------
bq. I can easily see us defaulting to such mechanism every time we get a flaky
test and that hiding genuine problems more often than not
If we, collectively, don't have the discipline not to abuse something like
this, we have bigger problems.
bq. this feels to me like the test isn't precise enough and should be improved
I'll let [~blambov] speak to that since this ticket was create from discussion
with him.
bq. that should be exceptional and I'd rather handle that in a case-by-case
basis
This ticket is about introducing infrastructure to allow us to formalize
handling that on a case-by-case basis. If we leave a precise comment in the
code, we still have a non-green test-board and the cognitive burden of
filtering out "known flaky" failures when checking test results.
> Extend testing infrastructure to handle expected intermittent flaky tests -
> see ReplicationAwareTokenAllocatorTest.testNewCluster
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-12277
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12277
> Project: Cassandra
> Issue Type: Bug
> Reporter: Joshua McKenzie
> Assignee: Branimir Lambov
> Priority: Minor
> Labels: test
>
> From an offline discussion:
> bq. The ReplicationAwareTokenAllocatorTest.testNewCluster failure is a flake
> -- randomness will sometimes (on the order of 1/100) cause it to fail.
> Extending the ranges to avoid these flakes goes too far and makes the test
> meaningless.
> bq. How about instead of @flaky/@Ignore which currently indicates a test that
> intermittently fails but we do not expect it to, we instead use @tries, or
> @runs, or some annotation that indicates "run this thing N times, if M pass
> we're good". This would allow us to keep the current "we don't care about
> these test results (in context of green test board) because intermittent
> failures are not expected and the test quality needs shoring up" from "we
> expect this test to fail sometimes in this particular way."
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
