[
https://issues.apache.org/jira/browse/CASSANDRA-12277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15397334#comment-15397334
]
Sylvain Lebresne commented on CASSANDRA-12277:
----------------------------------------------
Mostly lgtm with the following minor remarks:
* I really think we need more explanation on why it's hard to fix the
underlying problem. I'm sure it's obvious to you why randomness can cause the
test to fail, why it's hard to fix it, and why the test is still valuable
despite this, but it's not at all to me. I'm basically asking for some
explanation of what the test does, where is the randomness involved and why
it's not easy to control it better.
* A comment on {{flakyTest()}} explaining what it does, and hence why you
shouldn't abuse it and should always justify it's use, wouldn't be superfluous
imo.
* I believe we should be using {{%n}} instead of {{\n}} in format string for
cross-platform compatibility.
* If the test failed after the first try, since we're running it as many time
as asked anyway, I'd wrote a message on stderr for how many failures (over how
many runs) we had (I think you could infer that from the stack trace since
every failure is added as a suppressed exception, but a simple message is a bit
more user friendly).
> Extend testing infrastructure to handle expected intermittent flaky tests -
> see ReplicationAwareTokenAllocatorTest.testNewCluster
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-12277
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12277
> Project: Cassandra
> Issue Type: Bug
> Reporter: Joshua McKenzie
> Assignee: Branimir Lambov
> Priority: Minor
> Labels: test
>
> From an offline discussion:
> bq. The ReplicationAwareTokenAllocatorTest.testNewCluster failure is a flake
> -- randomness will sometimes (on the order of 1/100) cause it to fail.
> Extending the ranges to avoid these flakes goes too far and makes the test
> meaningless.
> bq. How about instead of @flaky/@Ignore which currently indicates a test that
> intermittently fails but we do not expect it to, we instead use @tries, or
> @runs, or some annotation that indicates "run this thing N times, if M pass
> we're good". This would allow us to keep the current "we don't care about
> these test results (in context of green test board) because intermittent
> failures are not expected and the test quality needs shoring up" from "we
> expect this test to fail sometimes in this particular way."
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
