ThreeFire1 opened a new issue, #18707: URL: https://github.com/apache/dolphinscheduler/issues/18707
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues. ### What happened During database failover testing on a 3.4.3 cluster, I observed a task finish while the database was temporarily unwritable. After database writes recovered, the task's database state was still `RUNNING_EXECUTION` during the observation period. While investigating this, I reproduced the following failure path on `dev`. It is consistent with the cluster symptom, although the available cluster logs do not establish the complete sequence of the failed update, retransmission, and ACK for that incident. `persistentTaskInstanceSuccessEventToDB()` changes the shared task instance to `SUCCESS` before calling `taskInstanceDao.updateById()`. If that call throws, the in-memory state remains `SUCCESS`. When the success event is retried, the state machine selects `TaskSuccessStateAction`, which skips processing the event. The handler can then send an ACK without retrying the database update. The database state stays `RUNNING_EXECUTION`, and the normal workflow topology transition for that success event is not published. ### What you expected to happen After a database update throws, the task should remain eligible to process a retried success event. Once persistence succeeds, the master should publish the workflow topology event and acknowledge the report. ### How to reproduce I reproduced this in a local regression test on `dev` commit `fa0fe2eab93abc2032af9558f1a7ffb6548152c8`, using the real task state actions and success event handler, with a mocked DAO and a separate persisted-state snapshot: 1. Start with a task in `RUNNING_EXECUTION` in both memory and the persisted snapshot. 2. Deliver a `TaskSuccessLifecycleEvent`. Make the first DAO update throw `DataAccessResourceFailureException` without updating the snapshot. 3. Allow subsequent DAO updates to succeed and deliver the same success event again. 4. The second delivery does not call the DAO again. The persisted snapshot is still `RUNNING_EXECUTION`, but the success report is acknowledged. The regression tests also exercise retry through `WorkflowEventBusFireWorker`. A Master integration test, `TaskSuccessPersistenceRecoveryTestCase`, uses the existing Spring Master test environment and an isolated H2 database. It starts a serial workflow A → B and injects a single `DataAccessResourceFailureException` before A's first success update. All other DAO calls use the real implementation. With the fix, the Master retries the update, both task rows become `SUCCESS`, and the workflow completes without restarting the Master or rerunning the workflow. On the unmodified code, the same integration test fails: the retry logs that A is already `SUCCESS`, and the workflow remains `RUNNING_EXECUTION` until the test times out. Both versions were checked with clean builds. The local integration test uses real H2 persistence and injects the exception at the DAO boundary to control the failure timing. It complements the earlier cluster observation; it does not establish how the database driver translated the error in that incident. ### Anything else This is related to #10854, fixed by #10866 in the old `TaskResultEventHandler`. That fix restored the task instance when persistence failed. The replacement state-machine path introduced in #16327 updates the shared state before persistence without that recovery behavior. I traced this relationship in the source; I did not run every historical version. The fix persists a copy first and updates the shared task instance after the DAO call returns normally. It does not change the handling of a DAO call that returns `false`. I have a fix ready with regression tests that fail before the fix and pass after it, including the Master integration test above. I'd like to submit a PR for this. ### Version dev ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
