yaodongen commented on issue #18624:
URL:
https://github.com/apache/dolphinscheduler/issues/18624#issuecomment-5594807753
Good questions — I dug into both.
### Why it did not shut itself down
It never disconnected, so that path was never reached. `RegistryStoppable` →
`System.exit(1)` fires on registry *connection loss*; here the JDBC registry
was healthy throughout. The master went STAND_BY from an ordinary `REMOVE`
event on `/nodes/master-coordinator`, not from an expired session.
The trigger is that `serverIdentify` is `masterConfig.getMasterAddress()`,
which on a StatefulSet is stable across restarts:
1. We restarted both masters. The previous process's ephemeral coordinator
node was still present, holding the **same** address string.
2. The new process ran `participateElection()`:
```java
if (registry.acquireLock(electionLock)) {
if (!registry.exists(selectorPath)) {
registry.put(selectorPath, serverIdentify, true);
return true;
}
return serverIdentify.equals(registry.get(selectorPath)); // true --
and no put()
}
```
The path existed and its value equalled its own identify, so it returned
`true` and became ACTIVE **without ever registering an ephemeral node of its
own**.
3. The dead process's client row was then purged and its ephemeral node
deleted. The resulting `REMOVE` carried that same address, which the live
master reads as "that was me" and goes STAND_BY.
master-0, where the two `serverStartupTime` values are the old and new
process at the same address:
```
03:24:39.332 WorkflowSerialCoordinator started...
03:24:40.374 Server
MasterServerMetadata(...serverStartupTime=1787709361200...) removed
03:24:40.376 Server
MasterServerMetadata(...serverStartupTime=1788837878575...) added
03:24:58.479 The status is standby now.
```
master-1 went through the same thing slightly earlier, so when it picked the
role back up it was on its second `changeToActive()` in one JVM:
```
03:23:54.550 TaskGroupCoordinator closed # -> STAND_BY
03:24:57.606 The status is active now.
03:24:57.607 WorkflowSerialCoordinator starting...
03:24:57.607 ERROR AbstractHAServer - Trigger ServerStatusChangeListener
from STAND_BY -> ACTIVE error
java.lang.IllegalStateException: InternalThread is already
started
```
### On #18416
I don't think it covers this. That one handles a heartbeat row purged while
the database is unavailable, so the client enters the disconnect state machine
and the service terminates. Here the database was fine and no session expired —
the flip came from a stale ephemeral node with a reused identity after a
restart, which #18416 would not have prevented.
More to the point, the coordinator defect is independent of *how* the flip
happens. Any `ACTIVE -> STAND_BY -> ACTIVE` within one JVM breaks it, including
a perfectly legitimate failover (peer dies, this master takes over, later steps
down, later takes over again), because `close()` leaves `internalThread` set.
`TaskGroupCoordinator.close()`, invoked from the same listener one line
earlier, already nulls it — this PR just makes the two match.
### Is this production
Our internal dev/staging cluster on EKS, not customer-facing. But the
workflows are our real ETL pipeline, and instances sat in `SERIAL_WAIT` for 6h+
before anyone noticed: the master keeps reporting ACTIVE and keeps holding the
leader node, so nothing alerts and no peer takes over.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
