C. Scott Andreas created CASSANDRA-21506:
--------------------------------------------
Summary: Avoid Python dtest failure cascade by validating/awaiting
bind port availability
Key: CASSANDRA-21506
URL: https://issues.apache.org/jira/browse/CASSANDRA-21506
Project: Apache Cassandra
Issue Type: Bug
Components: Test/dtest/python
Reporter: C. Scott Andreas
Failures in the Python dtest suite can cascade in cases where one launched
cluster has not fully been torn down or released its bind socket. In a recent
CI run, this resulted in 28 Python dtest failures requiring triage when one
would have sufficed.
In this scenario, node startup fails with:
{{{}IllegalStateException: A node with address /127.0.0.2:7000 already exists,
cancelling join{}}}(TCM {{{}Register.register{}}}), and every subsequent test
on that worker then failed on teardown ({{{}fixture_dtest_setup{}}}
AttributeErrors / "Unexpected error found in node logs").
ccm assigns every cluster the same loopback addresses
({{{}127.0.0.1/.2/.3{}}}on storage port {{{}7000{}}}). If a node from a
previous test has not fully released its address, or a concurrent worker uses
the same range, the next cluster's joining node contacts a seed that already
knows its address and aborts — cascading across the remaining tests on the
worker.
*Approach*
We can avoid this cascade by awaiting port availability for a short time.
Before a fresh cluster starts (in {{{}fixture_dtest_setup{}}}, after
{{{}initialize_cluster{}}}), waiting for {{127.0.0.1-4:7000}} to be free can
avoid failure cascade. This absorbs a slow-releasing prior node and serializes
an accidental overlap so both runs can pass, then fails fast with one clear
message if the ports do not clear within the timeout. When ports are already
free (normal case) it is a no-op.
Potential patch (model-assisted):
[https://github.com/cscotta/cassandra-dtest/pull/3/changes/b151c9094c485d4cdcac71cc02a060d5de728fb7]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
