[
https://issues.apache.org/jira/browse/CASSANDRA-12277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15391446#comment-15391446
]
Sylvain Lebresne commented on CASSANDRA-12277:
----------------------------------------------
I'll admit that at face value that seems a weird and potentially dangerous
thing to do to me. I can easily see us defaulting to such mechanism every time
we get a flaky test and that hiding genuine problems more often than not. I
don't know the details about that specific
{{ReplicationAwareTokenAllocatorTest.testNewCluster}} test, but if a test
knowingly fails a number of the times, this feels to me like the test isn't
precise enough and should be improved.
Note that I can buy that exceptionally a test is such that making it pass 100%
of the time is too much work to be worth the trouble but the test is still nice
to have, but that should be exceptional and I'd rather handle that in a
case-by-case basis, with very precise comment as to why this is allowed to fail
some of the times, to make it clear it's a special case.
> Extend testing infrastructure to handle expected intermittent flaky tests -
> see ReplicationAwareTokenAllocatorTest.testNewCluster
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-12277
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12277
> Project: Cassandra
> Issue Type: Bug
> Reporter: Joshua McKenzie
> Assignee: Branimir Lambov
> Priority: Minor
> Labels: test
>
> From an offline discussion:
> bq. The ReplicationAwareTokenAllocatorTest.testNewCluster failure is a flake
> -- randomness will sometimes (on the order of 1/100) cause it to fail.
> Extending the ranges to avoid these flakes goes too far and makes the test
> meaningless.
> bq. How about instead of @flaky/@Ignore which currently indicates a test that
> intermittently fails but we do not expect it to, we instead use @tries, or
> @runs, or some annotation that indicates "run this thing N times, if M pass
> we're good". This would allow us to keep the current "we don't care about
> these test results (in context of green test board) because intermittent
> failures are not expected and the test quality needs shoring up" from "we
> expect this test to fail sometimes in this particular way."
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
