WXPNG opened a new issue, #12734:
URL: https://github.com/apache/seatunnel/issues/12734
When a worker node (lite member) is started before the master node, the
worker cannot join the
cluster and exits after ~300s with `Could not join cluster. Shutting down
now!`.
Root cause: a worker is a Hazelcast lite member and can never become master
itself. When it starts
first, connecting to the not-yet-started masters fails, and the master
addresses get temporarily
blacklisted (`blacklist(address, false)`). Once blacklisted, the worker
never re-connects to them,
which deadlocks the join until the join timeout
(`hazelcast.max.join.seconds`).
Here is our current fix. Is it acceptable to the community, or is there a
better approach?
**1. Add a configurable retry interval in `SeaTunnelProperties`:**
```java
public static final HazelcastProperty LITE_NODE_JOIN_RETRY_INTERVAL_SECONDS =
new HazelcastProperty(
"seatunnel.lite.node.join.retry.interval.seconds", 5,
TimeUnit.SECONDS);
```
**2. Read it in `LiteNodeDropOutTcpIpJoiner` (field + constructor):**
```java
private final long liteNodeJoinRetryIntervalMillis;
// in constructor
liteNodeJoinRetryIntervalMillis =
node.getProperties()
.getMillis(SeaTunnelProperties.LITE_NODE_JOIN_RETRY_INTERVAL_SECONDS);
```
**3. Recover from the blacklist deadlock in `joinViaPossibleMembers()`:**
```java
// after tryClaimMastership(possibleAddresses), before
setMasterAddressToJoin(null)
if (node.isLiteMember() && isAllBlacklisted(possibleAddresses)) {
try {
clusterService.blockOnJoin(liteNodeJoinRetryIntervalMillis);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
for (Address address : possibleAddresses) {
unblacklist(address);
}
}
```
With this change, a worker started before the master waits for a retry
interval, clears the
temporary blacklist, and re-connects on the next `tryJoinAddresses()`. It
automatically joins the
cluster once the master comes up, while the overall
`hazelcast.max.join.seconds` timeout is kept
unchanged.
Is this modification acceptable, and would the community accept a PR for it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
