yaodongen opened a new pull request, #18623:
URL: https://github.com/apache/dolphinscheduler/pull/18623
## Was this PR generated or assisted by AI?
YES — the diagnosis, the patch and this description were written with AI
assistance (Claude). The root cause was confirmed against a live 3.4.2 cluster:
the stack trace, the `starting...`/`started...` mismatch and the recovery
behaviour below are all taken from that cluster's logs, not inferred.
## Purpose of the pull request
`WorkflowSerialCoordinator` is single-use, but the master restarts it on
every HA transition. After an `ACTIVE -> STAND_BY -> ACTIVE` flip **within the
same JVM**, `start()` throws and serial dispatch dies silently — `SERIAL_WAIT`
/ `SERIAL_DISCARD` / `SERIAL_PRIORITY` workflows then queue in
`t_ds_serial_command` forever.
`close()` leaves `internalThread` non-null:
```java
@Override
public void close() {
flag = false;
}
```
while `start()` refuses to run while it is set:
```java
if (internalThread != null) {
throw new IllegalStateException("InternalThread is already started");
}
```
and `MasterCoordinator.MasterCoordinatorListener` calls both against the
same instance:
```java
public void changeToActive() { taskGroupCoordinator.start();
workflowSerialCoordinator.start(); ... }
public void changeToStandBy() { taskGroupCoordinator.close();
workflowSerialCoordinator.close(); ... }
```
The sibling `TaskGroupCoordinator.close()`, invoked from that same listener,
already interrupts the thread and nulls the field — which is why task groups
survive a failover and serial dispatch does not. This PR makes
`WorkflowSerialCoordinator.close()` match it.
Two details make this silent rather than loud, and are the reason it is
worth fixing rather than documenting:
- `AbstractHAServer.statusChange` assigns `serverStatus` **before**
notifying listeners and swallows listener exceptions. The master therefore ends
up reporting ACTIVE with a dead coordinator while still holding
`/nodes/master-coordinator`, so no peer takes over and nothing self-heals.
- `taskGroupCoordinator.start()` runs first and succeeds, so the master
keeps looking healthy.
The only external symptom is a `WorkflowSerialCoordinator starting...` log
line with no matching `started...`.
Observed on 3.4.2 (2 masters, JDBC registry): workflow instances stuck in
`SERIAL_WAIT` for 6h+, the oldest serial command waiting since 04:00 UTC,
`starting...` logged twice against `started...` once:
```
2026-09-08 03:24:57.607 INFO o.a.d.s.m.e.w.s.WorkflowSerialCoordinator -
WorkflowSerialCoordinator starting...
2026-09-08 03:24:57.607 ERROR o.a.d.r.a.h.AbstractHAServer - Trigger
ServerStatusChangeListener from STAND_BY -> ACTIVE error
java.lang.IllegalStateException: InternalThread is already started
at
org.apache.dolphinscheduler.server.master.engine.workflow.serial.WorkflowSerialCoordinator.start(WorkflowSerialCoordinator.java:83)
at
org.apache.dolphinscheduler.server.master.engine.MasterCoordinator$MasterCoordinatorListener.changeToActive(MasterCoordinator.java:98)
at
org.apache.dolphinscheduler.registry.api.ha.AbstractServerStatusChangeListener.change(AbstractServerStatusChangeListener.java:33)
```
Affects every 3.4.x — `WorkflowSerialCoordinator` was added in 3.4.0 by
#17531 (DSIP-92) and `close()` has not changed since; still present in 3.4.3
and on `dev`. Single-master deployments cannot hit it, because
`AbstractHAServer` only returns to ACTIVE from a REMOVE event carrying
*another* server's identify, which cannot occur when one master is the sole
writer of that path. That is presumably why it has gone unreported.
## Brief change log
- `WorkflowSerialCoordinator.close()` now interrupts `internalThread` and
sets it to `null`, returns early when already closed, and is `synchronized` to
pair with the already-`synchronized` `start()` (`internalThread` is not
`volatile`). Line-for-line mirror of `TaskGroupCoordinator.close()`.
- Add `WorkflowSerialCoordinatorTest` covering `start -> close -> start`,
plus `start` while started and a repeated `close`.
`TaskGroupCoordinatorTest.start()` covers only `start -> close`, which is why
this asymmetry went unnoticed.
## Verify this pull request
Manually verified on a live 3.4.2 cluster:
- **Reproduced** — restarting both masters together produced the same-JVM
`ACTIVE -> STAND_BY -> ACTIVE` flip and the exception above; serial dispatch
stopped while the master still reported ACTIVE.
- **Confirmed the fix's premise** — bringing the coordinator up with
`internalThread == null` (the state this patch restores on close) starts it
cleanly and immediately drains the backlog:
```
WorkflowSerialCoordinator starting...
WorkflowSerialCoordinator started...
Launched SerialCommand: id=59, workflowInstanceId=69 # queued since 04:00
UTC
```
Built and tested locally on JDK 11 (one of the two CI targets; `pom.xml`
targets source/target 1.8).
**Without the fix** the new test reproduces the exact production failure —
same exception, same line:
```
Tests run: 3, Failures: 1, Errors: 0, Skipped: 0
[ERROR] WorkflowSerialCoordinatorTest.startAfterCloseShouldNotThrow <<<
FAILURE!
org.opentest4j.AssertionFailedError: Unexpected exception thrown:
java.lang.IllegalStateException: InternalThread is already started
Caused by: java.lang.IllegalStateException: InternalThread is already started
at ...WorkflowSerialCoordinator.start(WorkflowSerialCoordinator.java:83)
```
**With the fix:**
```
./mvnw -pl dolphinscheduler-master -am test
-Dtest=WorkflowSerialCoordinatorTest
...
[INFO] --- spotless-maven-plugin:2.27.2:check (default) @
dolphinscheduler-master ---
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS
```
`spotless:check` is bound to the `compile` phase and was not skipped, so
formatting is covered too.
This change is backwards compatible and needs no entry in
`docs/docs/en/guide/upgrade/incompatible.md`.
## Pull Request Notice
[Pull Request
Notice](https://github.com/apache/dolphinscheduler/blob/dev/docs/docs/en/contribute/join/pull-request.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
