michaellx1057 commented on PR #18645:
URL:
https://github.com/apache/dolphinscheduler/pull/18645#issuecomment-5696146905
@ruanwenjun
> Will there be any issues if the service restarts quickly and assumes the
role of the previous leader?
Thanks for asking. Yes—in our reproduction, the restarted Master became
ACTIVE without acquiring ownership of the existing coordinator selector. That
selector still belonged to the previous process's etcd lease and was
subsequently removed. A matching address does not transfer the old process's
lease to the new process.
### Deployment and operational context
We run a self-hosted DolphinScheduler 3.4.2 deployment with:
- Kubernetes v1.33.13: 6 nodes, including 3 control-plane nodes.
- DS Master: 2 replicas in a StatefulSet for availability and failover,
reusing stable addresses such as `ds-master-0.ds-master:5678` after restart.
- DS Registry: a separate 3-member etcd 3.6.14 cluster, distinct from
Kubernetes control-plane etcd.
We encountered scheduling problems during Master maintenance, restarts and
replica-count changes. The controlled reproduction kept DS at 3.4.2 throughout;
it did not involve a mixed-version upgrade. We temporarily reduced the
deployment to 1 Master afterward. The proposed HA patch has not been deployed
to this environment.
### Restarting a Master at the same address
In a separate controlled run on 2026-09-16 (all times below are UTC+8):
1. Restore the Master StatefulSet to 2 replicas and check both Pods.
2. Restart `ds-master-0` by deleting its Pod, allowing the StatefulSet to
recreate it at the same address.
3. Observe both Masters' role transitions and serial-coordinator logs
alongside the etcd selector's lease and watch events.
The recreated Master 0 became ACTIVE while the coordinator selector was
still attached to the predecessor's lease.
Both Masters logged ACTIVE and started their coordinators:
```text
# ds-master-0
[WI-0][TI-0] - 2026-09-16 15:08:47.609 INFO [Master-Server]
o.a.d.r.a.h.DefaultServerStatusChangeListener:[27] - The status is active now.
[WI-0][TI-0] - 2026-09-16 15:08:47.610 INFO [Master-Server]
o.a.d.s.m.e.w.s.WorkflowSerialCoordinator:[90] - WorkflowSerialCoordinator
started...
# ds-master-1
[WI-0][TI-0] - 2026-09-16 15:08:48.281 INFO [grpc-default-executor-0]
o.a.d.r.a.h.DefaultServerStatusChangeListener:[27] - The status is active now.
[WI-0][TI-0] - 2026-09-16 15:08:48.282 INFO [grpc-default-executor-0]
o.a.d.s.m.e.w.s.WorkflowSerialCoordinator:[90] - WorkflowSerialCoordinator
started...
```
The etcd watch recorded deletion of the old selector, followed by Master 1
writing a new selector. Selected decoded fields:
```text
DELETE /services/dolphinscheduler/nodes/master-coordinator
mod_revision: 4808188
value: ""
prev_value: ds-master-0.ds-master:5678
prev_lease: 2559627622716187248
prev_mod_revision: 4807750
PUT /services/dolphinscheduler/nodes/master-coordinator
mod_revision: 4808190
value: ds-master-1.ds-master:5678
lease: 2559627622716203084
```
More than 7 minutes later, both coordinators were still processing the same
workflow instance:
```text
# ds-master-0
[WI-0][TI-0] - 2026-09-16 15:16:18.357 INFO
[WorkflowSerialCoordinator-Thread]
o.a.d.s.m.e.w.s.SerialCommandPriorityHandler:[54] - Stop the pre
WorkflowInstance: 109018 due to the workflow using SERIAL_PRIORITY strategy
# ds-master-1
[WI-0][TI-0] - 2026-09-16 15:16:18.693 INFO
[WorkflowSerialCoordinator-Thread]
o.a.d.s.m.e.w.s.SerialCommandPriorityHandler:[54] - Stop the pre
WorkflowInstance: 109018 due to the workflow using SERIAL_PRIORITY strategy
```
There are recurring entries between these points, so this was sustained
coordinator overlap. These logs do not establish duplicate business writes. The
overlap also involved the etcd REMOVE payload issue and failure to demote after
losing an election; address reuse was the initial ownership problem, not the
entire failure chain.
### Related scale-down observation
In an earlier round, we scaled from 2 Masters to 1. This exposed the
separate REMOVE-event problem tracked in #18640/#18641; it is not the same
experiment as the restart above.
Before recovery, the database snapshot showed an unassigned workflow waiting
to run:
```text
# 2026-09-16 15:00:16.984562+08; selected fields from the database snapshot
id=111167, state=14 (SERIAL_WAIT), host=<empty>, end_time=<empty>
start_time=2026-09-16 15:00:00.030
```
The remaining Master logged:
```text
[WI-0][TI-0] - 2026-09-16 15:00:32.273 ERROR [grpc-default-executor-28]
o.a.d.s.m.c.AbstractClusterSubscribeListener:[36] - Unknown cluster change
event: Event(watchedPath=/nodes/master,
eventPath=/nodes/master/ds-master-1.ds-master:5678, eventData=, type=REMOVE)
```
We initiated a restart of the remaining Master at approximately 15:00:47,
using the same DS 3.4.2 image. The delayed instance then completed:
```text
[WI-111167][TI-0] - 2026-09-16 15:03:15.108 INFO
[ds-workflow-eventbus-worker-6]
o.a.d.s.m.e.w.s.AbstractWorkflowStateAction:[180] - Success set
WorkflowExecuteRunnable: realtime_compare_2m-20260916150000030 state from:
RUNNING_EXECUTION to SUCCESS
```
Database readback confirmed instance 111167 was SUCCESS at 15:03:15.105, and
the command queue was empty at 15:04:21. Instance 111168 also completed at
15:03:24.923. Their elapsed times were approximately 195 and 85 seconds,
compared with about 5 seconds for adjacent executions. These durations include
the recovery process.
This screenshot shows the final SUCCESS records after recovery. The
fault-time SERIAL_WAIT state is established by the database snapshot; we did
not capture a UI screenshot at that point.
### Supplementary MySQL check
We also tested the original HA implementation with a real MySQL 8.0.46
Registry. The successor became ACTIVE, but selector row 1 still belonged to the
predecessor client, with its value and update time unchanged. This was an
isolated fixture deliberately retaining the predecessor, not a production JDBC
restart.
This is why I think the new process should acquire its own selector rather
than adopt the predecessor's role based on its address. Adding an instance
timestamp addresses that ownership ambiguity. Handling a lost election and
stopping the old coordinator are separate parts of the fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
