michaellx1057 commented on PR #18645: URL: https://github.com/apache/dolphinscheduler/pull/18645#issuecomment-5699157892
I tested a smaller candidate to check which changes are needed for this fix. [3687df9](https://github.com/apache/dolphinscheduler/pull/18645/commits/3687df9cb819b5acb2bf6099b80f4f4094f36a5f) now narrows this PR to the instance timestamp, ownership-based role reconciliation, election/role-update synchronization, and the coordinator thread handoff protection from 48f3e2a. It removes immediate demotion on election exceptions, the `closed` handling, and the related Alert resource-cleanup changes. One clarification to my [earlier coordinator comment](https://github.com/apache/dolphinscheduler/pull/18645#issuecomment-5696147234): the overlap does not require the exception → standby → retry path. Without the handoff protection, tests also reproduced it during normal loss and reacquisition of leadership. A blocked DAO call was allowed to return after reactivation, and both the old and new threads continued polling. This is why I am keeping the handoff protection while removing the broader lifecycle changes. ### Validation The combined candidate passed 26 targeted tests on JDK 8. The final upstream revision was also clean-built through the 132-module Master/Alert reactor on actual Zulu JDK 8 `1.8.0_504`: all 36 distinct selected cases passed (13 HA, 6 serial-coordinator, 7 task-group-coordinator and 10 existing Alert cases), with no failures, errors or skips. We also tested a 3.4.2 backport in the same Kubernetes / etcd deployment described above, including a leader restart at the same address and scaling from 2 Masters to 1. The backport also covers the serial-coordinator restartability fix already present in dev (#18623). It is not a deployment of dev jars into 3.4.2. Selected fields from the validation logs follow. Times are UTC+8 on 2026-09-16; thread-dump details are omitted for readability. The thread lists contain only the serial and task-group coordinators. ```text 20:56:28 — 2 Masters ready selector = ds-master-0.ds-master:5678#1789563214717 ds-master-0 = [WorkflowSerialCoordinator-Thread, TaskGroupCoordinator-Thread] ds-master-1 = [] 21:01:36 — after restarting the active Master-0 selector = ds-master-1.ds-master:5678#1789563298359 ds-master-0 = [] ds-master-1 = [WorkflowSerialCoordinator-Thread, TaskGroupCoordinator-Thread] 21:06:11 — after scaling down and removing active Master-1 selector = ds-master-0.ds-master:5678#1789563580896 ds-master-0 = [WorkflowSerialCoordinator-Thread, TaskGroupCoordinator-Thread] ``` The rebuilt Master-0 remained standby while Master-1 owned the selector. After Master-1 was removed, Master-0 acquired the selector and started both coordinators. Natural scheduled executions also completed successfully. For example, selected database fields show: ```text id=111563 host=ds-master-1.ds-master:5678 state=7 (SUCCESS) start=21:04:00.051 end=21:04:06.056 id=111566 host=ds-master-0.ds-master:5678 state=7 (SUCCESS) start=21:06:00.008 end=21:06:03.848 ``` The error-command count remained at its baseline of 12 throughout the recorded checkpoints. The original image and single-Master topology were restored afterward. An earlier attempt restored the original deployment before completing the restart check because the test's 90-second wait was shorter than the configured 120-second termination grace period. The results above are from the subsequent run with the corrected timeout. These observations support the reduced scope for the reported restart and scale-down scenarios. The snapshots do not prove that no transient overlap occurred between observations, and the patch does not provide cross-Master fencing. The separate etcd REMOVE-value fix remains tracked in #18641. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
