[
https://issues.apache.org/jira/browse/CASSANDRA-21506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nivy Kani reassigned CASSANDRA-21506:
-------------------------------------
Assignee: Nivy Kani
> Avoid Python dtest failure cascade by validating/awaiting bind port
> availability
> --------------------------------------------------------------------------------
>
> Key: CASSANDRA-21506
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21506
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Test/dtest/python
> Reporter: C. Scott Andreas
> Assignee: Nivy Kani
> Priority: Normal
>
> Failures in the Python dtest suite can cascade in cases where one launched
> cluster has not fully been torn down or released its bind socket. In a recent
> CI run, this resulted in 28 Python dtest failures requiring triage when one
> would have sufficed.
> In this scenario, node startup fails with:
> {{{}IllegalStateException: A node with address /127.0.0.2:7000 already
> exists, cancelling join{}}}(TCM {{{}Register.register{}}}), and every
> subsequent test on that worker then failed on teardown
> ({{{}fixture_dtest_setup{}}} AttributeErrors / "Unexpected error found in
> node logs").
> ccm assigns every cluster the same loopback addresses
> ({{{}127.0.0.1/.2/.3{}}}on storage port {{{}7000{}}}). If a node from a
> previous test has not fully released its address, or a concurrent worker uses
> the same range, the next cluster's joining node contacts a seed that already
> knows its address and aborts — cascading across the remaining tests on the
> worker.
> *Approach*
> We can avoid this cascade by awaiting port availability for a short time.
> Before a fresh cluster starts (in {{{}fixture_dtest_setup{}}}, after
> {{{}initialize_cluster{}}}), waiting for {{127.0.0.1-4:7000}} to be free can
> avoid failure cascade. This absorbs a slow-releasing prior node and
> serializes an accidental overlap so both runs can pass, then fails fast with
> one clear message if the ports do not clear within the timeout. When ports
> are already free (normal case) it is a no-op.
> Potential patch (model-assisted):
> [https://github.com/cscotta/cassandra-dtest/pull/3/changes/b151c9094c485d4cdcac71cc02a060d5de728fb7]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
