yaodongen opened a new issue, #18624: URL: https://github.com/apache/dolphinscheduler/issues/18624
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues. ### What happened `SERIAL_WAIT` / `SERIAL_DISCARD` / `SERIAL_PRIORITY` workflows stop being dispatched after the master coordinator changes hands, and never recover. The master still reports itself ACTIVE and holds the leader node, so no peer takes over either. The only trace is a single swallowed exception: ``` 2026-09-08 03:24:57.607 INFO o.a.d.s.m.e.w.s.WorkflowSerialCoordinator - WorkflowSerialCoordinator starting... 2026-09-08 03:24:57.607 ERROR o.a.d.r.a.h.AbstractHAServer - Trigger ServerStatusChangeListener from STAND_BY -> ACTIVE error java.lang.IllegalStateException: InternalThread is already started at org.apache.dolphinscheduler.server.master.engine.workflow.serial.WorkflowSerialCoordinator.start(WorkflowSerialCoordinator.java:83) at org.apache.dolphinscheduler.server.master.engine.MasterCoordinator$MasterCoordinatorListener.changeToActive(MasterCoordinator.java:98) at org.apache.dolphinscheduler.registry.api.ha.AbstractServerStatusChangeListener.change(AbstractServerStatusChangeListener.java:33) ``` Note the `starting...` with no matching `started...` — that mismatch is the only way to detect the state from outside. **Root cause.** `WorkflowSerialCoordinator.close()` does not reset `internalThread`: ```java @Override public void close() { flag = false; } ``` while `start()` refuses to run if it is still set: ```java public synchronized void start() { if (flag) { throw new IllegalStateException("WorkflowSerialCoordinator is already started"); } if (internalThread != null) { throw new IllegalStateException("InternalThread is already started"); } ... } ``` So the coordinator is single-use, but `MasterCoordinatorListener` treats it as restartable — `changeToStandBy()` calls `close()` and `changeToActive()` calls `start()` on the same instance: ```java public void changeToActive() { taskGroupCoordinator.start(); workflowSerialCoordinator.start(); ... } public void changeToStandBy() { taskGroupCoordinator.close(); workflowSerialCoordinator.close(); ... } ``` The sibling `TaskGroupCoordinator.close()` in that same listener already handles this correctly (interrupts the thread and nulls the field), which is why task groups survive a failover and serial dispatch does not. Two things make the failure silent rather than loud: 1. `AbstractHAServer.statusChange` sets `serverStatus` **before** notifying the listeners, and swallows listener exceptions: ```java private void statusChange(ServerStatus targetStatus) { final ServerStatus originStatus = serverStatus; serverStatus = targetStatus; // already ACTIVE synchronized (this) { try { serverStatusChangeListeners.forEach(listener -> listener.change(originStatus, serverStatus)); } catch (Exception ex) { log.error("Trigger ServerStatusChangeListener from {} -> {} error", originStatus, targetStatus, ex); } } } ``` The master therefore ends up ACTIVE with a dead coordinator, holding `/nodes/master-coordinator`, so no other master will take over. 2. `taskGroupCoordinator.start()` runs first and succeeds, so task-group scheduling keeps working and the master looks healthy. ### What you expected to happen After an `ACTIVE -> STAND_BY -> ACTIVE` transition the serial coordinator restarts and queued `t_ds_serial_command` rows are dispatched, as they are on a fresh master process. ### How to reproduce 1. Run two masters with the JDBC registry. 2. Create a workflow whose execution type is `SERIAL_WAIT` and put it on a schedule. 3. Make the coordinator change hands twice, so that one master goes `ACTIVE -> STAND_BY -> ACTIVE` within the same JVM. Restarting both masters together is enough; so is any event that reaps the leader's ephemeral node (with `REGISTRY_TERM_REFRESH_INTERVAL=2s` and `REGISTRY_TERM_EXPIRE_TIMES=3`, a 6s stall does it). 4. That master logs the exception above. 5. `t_ds_serial_command` rows stay at `state=0` (WAITING) indefinitely and workflow instances stay in `SERIAL_WAIT`. Observed on 3.4.2: instances queued for 6h+, the oldest serial command waiting since 04:00 UTC. Restarting the ACTIVE master (fresh JVM, `internalThread` null) recovers immediately — the queued commands launch within one 5s round. A unit test reproduces it directly: `start()` -> `close()` -> `start()` on one instance throws `InternalThread is already started`. ### Anything else - Affects every 3.4.x. `WorkflowSerialCoordinator` was introduced in 3.4.0 by #17531 (DSIP-92, refactor workflow serial strategy) and `close()` has been unchanged since. Still present on `dev` and in 3.4.3. - Single-master deployments never hit it: `AbstractHAServer` only returns to ACTIVE from a REMOVE event carrying *another* server's identify, which cannot occur when one master is the sole writer of that path. That is presumably why this has gone unreported. - Workaround until the fix ships: run a single master, or restart the ACTIVE master after any failover and confirm `starting...`/`started...` are paired. ### Version 3.4.2 ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
