michaellx1057 opened a new issue, #18640:
URL: https://github.com/apache/dolphinscheduler/issues/18640
### Search before asking
- [x] I searched the existing issues and pull requests and did not find a
duplicate report for the etcd-specific DELETE conversion described below.
Related: #18296 fixes a similar missing-value problem in the **JDBC**
registry.
The etcd implementation still reads the current DELETE KV rather than its
previous value.
### What happened
With DolphinScheduler 3.4.2 and the etcd registry, a removed Master can
remain
in another Master's in-memory membership list. After reducing a two-Master
cluster to one Master, the remaining process can continue using two command
slots, leaving commands assigned to the departed Master unconsumed.
The registry contains only the live Master, but the subscriber rejects the
removal event because its heartbeat data is empty. A representative,
sanitized message is:
```text
Unknown cluster change event: Event(..., eventData=, type=REMOVE)
```
#### Root cause
`EtcdRegistry.subscribe()` already requests `withPrevKV(true)`. However,
`EtcdRegistry.toEvent()` maps DELETE to REMOVE and then reads `eventData`
from `watchEvent.getKeyValue().getValue()` for every event type.
For an etcd DELETE, the current KV identifies the deleted key and deletion
revision; the deleted value is available in `prev_kv`. Consequently, the
converted REMOVE has an empty value even though the previous value was
provided by the watch.
`AbstractClusterSubscribeListener.notify()` parses the heartbeat before
dispatching REMOVE and returns when parsing yields null. The departed Master
therefore remains in `MasterClusters`. Updates for the surviving Master do
not remove that entry. With both cached members NORMAL, command partitioning
can remain at two slots.
- [etcd event semantics](https://etcd.io/docs/v3.5/learning/api/#events)
- [EtcdRegistry conversion, inspected
dev](https://github.com/apache/dolphinscheduler/blob/9839c418c1a6d7f2a8c3395552edcd99c2de1b37/dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-etcd/src/main/java/org/apache/dolphinscheduler/plugin/registry/etcd/EtcdRegistry.java#L419)
- [Cluster subscriber, inspected
dev](https://github.com/apache/dolphinscheduler/blob/9839c418c1a6d7f2a8c3395552edcd99c2de1b37/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/cluster/AbstractClusterSubscribeListener.java#L29)
### What you expected to happen
An etcd REMOVE event should carry the deleted node's previous value and the
deleted key's path. Subscribers should be able to remove the departed member
and recalculate the surviving Master's command slot.
ADD and UPDATE should continue carrying the current value.
### How to reproduce
#### Minimal event reproduction
Use an isolated local etcd instance. In one terminal, start the watch:
```bash
etcdctl --endpoints=http://127.0.0.1:2379 \
watch /ds-repro/member --prev-kv --write-out=json
```
In another terminal:
```bash
etcdctl --endpoints=http://127.0.0.1:2379 \
put /ds-repro/member heartbeat-before-delete
etcdctl --endpoints=http://127.0.0.1:2379 \
del /ds-repro/member
```
The DELETE event's current `kv` has no value, while `prev_kv` contains
`heartbeat-before-delete`. This was reproduced with local etcd 3.5.21.
Constructing the corresponding jetcd 0.5.11 `WatchEvent` and passing it to
the deployed 3.4.2 converter yields REMOVE with empty `eventData`.
Using the deployed `MasterClusters`, an empty-data REMOVE leaves both
members present; supplying the deleted member's previous heartbeat removes
it. The local regression tests accompanying the proposed fix cover this
event-conversion contract without requiring Docker.
The focused test is `EtcdRegistryEventTest.testDeleteUsesPreviousValue`;
`testAddUsesCurrentValue`, `testUpdateUsesCurrentValue`, and
`testDeleteWithoutPreviousValuePreservesPath` cover the corresponding
controls.
With the regression test added and the production converter left unchanged,
run the focused reactor test with JDK 8:
```bash
./mvnw -pl
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-etcd
\
-am test -Dtest=EtcdRegistryEventTest \
-Dsurefire.failIfNoSpecifiedTests=false
```
The original converter fails `testDeleteUsesPreviousValue`: expected
`previous-heartbeat`, actual empty string. The three control tests pass.
#### Master-level reproduction conditions
In an isolated DS deployment:
1. Use the etcd registry and two NORMAL Masters. Confirm that the surviving
Master has cached both members.
2. Stop the other Master and let its membership key be deleted. Keep the
surviving JVM running.
3. Check the surviving Master's log for the empty REMOVE above.
4. With `idStep=1`, observe that commands assigned to the departed slot can
remain pending while commands assigned to the live slot are consumed.
Restarting the surviving JVM rebuilds membership and can clear this symptom;
it does not fix the conversion defect. If the departed cached member was
already BUSY, or is replaced at the same address, the slot symptom may
differ.
### Anything else
#### Affected versions checked
- Deployed and locally tested: 3.4.2.
- Same event conversion in 3.4.3, commit
`a190201acffa03d199d4ca216288734a6513de3d`.
- Same event conversion in dev, commit
`9839c418c1a6d7f2a8c3395552edcd99c2de1b37`.
#### Proposed fix
Keep `eventPath` derived from the current KV, and use `prevKV.value` for
DELETE `eventData`. Keep the current value for PUT events. The watch already
requests previous KVs, so this needs no configuration change or dependency.
Add focused tests for DELETE previous-value delivery, current-key retention,
and unchanged ADD/UPDATE behavior. This issue concerns event data and stale
membership. It does not claim to resolve every coordinator failover or
duplicate workflow execution problem.
#### Patch validation
The proposed patch passes all 4 event-conversion tests with JDK 8.
A clean reactor run including the existing etcd registry and lease tests
passes 18 tests with no failures, errors, or skipped tests. Spotless checks
also pass. This is module-level validation, not a patched multi-Master
end-to-end or production deployment test.
#### 中文说明
在使用 etcd 的 DS 3.4.2 中，退出 Master 的 REMOVE 事件可能被存活 Master
忽略，导致内存成员表仍保留旧节点，部分命令因分片残留无法消费。
根因是 watch 已开启 `prevKV`，但 DELETE 转换仍从当前 KV 读取 value。
etcd 的 DELETE 当前 KV 不含旧值，旧值在 `prev_kv`；因此事件数据为空，
成员监听器解析心跳后提前返回，未执行成员删除。
该行为已通过真实本地 etcd 事件和部署的 3.4.2 类验证。3.4.3 与上述固定 dev
提交仍有同样代码。建议仅在 DELETE 时使用旧值，并保留当前 key，增加回归测试。
修复后的 JDK 8 clean 构建共 18 项测试通过，格式检查通过；尚未部署或进行多 Master 端到端验收。
已有 #18296 修复的是 JDBC 注册中心，不能据此认定 etcd 已修复。
复现成员残留时需要两台 NORMAL Master，退出其中一台，保持另一台 JVM 不重启；
`idStep=1` 时可观察到对应分片命令积压。最小 etcd 命令见上方复现步骤。
本 issue 不将其它双协调器或重复执行问题一并声明为已定位或已解决。
#### AI assistance
Codex generated the initial proposed patch and regression tests, assisted
with source analysis and this report, and ran local reproduction and test
commands. Reproduction evidence and investigation limits are described above.
Codex 生成了初始修复与回归测试，协助源码分析和本报告，并执行了本地复现及测试命令。
复现证据和调查边界见上文。
### Version
3.4.2
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's
[Code of Conduct](https://www.apache.org/foundation/policies/conduct).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail:
[email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
