michaellx1057 opened a new issue, #18644: URL: https://github.com/apache/dolphinscheduler/issues/18644
## Search before asking - [x] I searched existing issues; related reports and the distinct remaining defect are described below. This is a focused follow-up to the stale-identity trigger already described by @yaodongen in [#18624](https://github.com/apache/dolphinscheduler/issues/18624#issuecomment-5594807753). That issue was closed by #18623, which fixes serial coordinator restartability. This report concerns the remaining shared `AbstractHAServer` ownership and role-reconciliation logic. Related etcd adapter issue #18640 and PR #18641 remain separate. ## What happened A restarted HA server can enter ACTIVE while its predecessor's ephemeral selector still exists. `serverIdentify` is the stable service address, and `participateElection()` accepts an existing value equal to that address without creating a selector owned by the new process. After the predecessor's key disappears, the new process can lose another election but retain ACTIVE: the non-self REMOVE branch applies ACTIVE on success and does not apply STAND_BY on failure. Startup election and status application also run independently of watch callbacks, allowing local results and transitions to be applied out of order. We reproduced the persistent dual-coordinator consequence on DS 3.4.2 with etcd using an ordinary StatefulSet restart. The production store was etcd 3.6.14; a separate isolated probe also reproduced the DS path with 3.5.21. Both versions define the preceding value in `prev_kv`, so the evidence does not point to a version-specific etcd defect. The store had a single selector owner; two DS serial coordinators continued scanning. The etcd adapter's empty REMOVE value contributes to this particular path. No broken etcd lock or namespace split was required. We also reproduced the common ownership defect first-hand against all three Registry implementations using independent predecessor and successor clients: | Real backend | Original HA source | Proposed fix | Direct ownership check | | --- | --- | --- | --- | | etcd 3.5.21 | ACTIVE | STAND_BY | Predecessor lease, modRevision and value unchanged | | ZooKeeper 3.8.3 / Curator 5.5.0 | ACTIVE | STAND_BY | Distinct sessions; predecessor ephemeralOwner and mzxid unchanged | | JDBC / MySQL 8.0.46 | ACTIVE | STAND_BY | Distinct client IDs; predecessor row ID, client_id, value and update time unchanged | | JDBC / PostgreSQL 16.0 | ACTIVE | STAND_BY | Distinct client IDs; predecessor row ID, client_id, value and update time unchanged | All eight original/fixed cases passed their respective assertions. In both JDBC databases, deleting the predecessor selector through its Registry client subsequently delivered the real change event: the fixed successor became ACTIVE and created a selector owned by its own client ID. These two recovery checks also passed. These fixtures deliberately retain the predecessor session/lease/client and seed a legacy bare-address selector. They establish the shared identity error; they do not reproduce a killed JVM, natural expiration, complete Master startup or duplicate workflows on every backend. The ZooKeeper/etcd probes use frozen HA/plugin sources and mixed cached supporting dependencies, not an official release distribution. JDBC uses the compiled dev classes, native database schemas and real Spring/MyBatis Registry wiring. The predecessor-identity trigger was already reported for JDBC in #18624; this report credits that analysis and adds cross-backend verification. ## What you expected to happen - A fresh HA instance must not adopt a predecessor's selector merely because its address is unchanged. - Current ownership should determine the local role after an election; losing or failing to establish ownership must not leave the server ACTIVE. - A delayed notification must not overwrite a newer local role decision or force a needless restart when the current selector is still owned by this instance. ## How to reproduce Common scenario reproduced against each real backend: 1. Keep a predecessor's ephemeral selector with value `master-a:5678` in the Registry. 2. Create a new `AbstractHAServer` instance with the same address and call `start()`. 3. Observe ACTIVE without a new `put()`; the predecessor still owns the node. Additional etcd consequence, rather than a claim about every backend: 4. Delete the predecessor's selector and allow another HA server to create its selector first. 5. Deliver the removal to the first server with empty event data, as the current etcd adapter does. 6. It loses the election but stays ACTIVE, while the peer is also ACTIVE. Deployment observation, DS 3.4.2 / etcd, times UTC+8: ```text 15:08:47.609 restarted master-a reports ACTIVE 15:08:47.610 master-a serial coordinator started 15:08:48.xxx predecessor lease expires; selector removed 15:08:48.281 master-b reports ACTIVE after writing the new selector 15:08:48.282 master-b serial coordinator started Both coordinators keep scanning for more than seven minutes. ``` This captured window proves concurrent coordinators, not duplicate creation of a new command: only master-a emitted new serial launch records in that window. Complete business logs contain private parameters and are not attached. ## Proposed fix / discussion Use a unique per-instance election identity, reconcile current ownership for every REMOVE event, and serialize local election and role application. Add backend-independent HA tests covering same-address restart, failed election, delayed/empty REMOVE and startup/watch ordering. Respect terminal consumer shutdown: Alert closes the service on demotion, so a later retry must not reactivate it. The selector value would become an opaque address-plus-UUID identity. In-tree consumers compare it as an opaque string. External tools that parse it as a bare address need an upgrade note. Mixed-version peers can recognize it as a different contender, but unpatched peers retain their existing defects. This is not a fencing protocol and does not promise zero overlap during watch delays, network partitions or downstream work already in progress. Those require separate guarantees. ## Version Reproduced on 3.4.2. The relevant shared HA implementation is unchanged in 3.4.3 and dev at `9839c418c1a6d7f2a8c3395552edcd99c2de1b37`. ## 中文说明 这是对 #18624 已有身份复用分析的后续修复，不是重复报告其已经修好的串行线程重启问题。新进程仅因地址相同就继承旧进程的协调器身份；删除事件触发再次参选后，即使落选，也可能保留 ACTIVE。我们在 etcd 上复现了持续双协调器，本地真实 etcd、ZK、JDBC（MySQL 和 PostgreSQL）新旧客户端对照均确认了身份误认这一层，补丁均拒绝误认；两个 JDBC 后端还验证了旧节点删除后可以正常接管。不能据此声称所有 Registry 都已复现相同双活后果。 建议在共享 HA 层区分进程实例、统一按当前所有权更新角色，并增加可确定重现竞态的通用测试。现有 #18641 仍负责正确传递 etcd 删除事件旧值；两项修复应独立审阅。补丁不包含分布式 fencing，不将修复范围夸大为消除所有分区或通知延迟下的重叠执行。 ## Contribution A focused shared-layer fix and regression tests have been prepared with Codex assistance and will be submitted as a separate PR. The patch has not been deployed to production. We welcome review of the identity-value compatibility and reconciliation approach. ## Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ## Code of Conduct - [x] I agree to follow the project [Code of Conduct](https://www.apache.org/foundation/policies/conduct). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
