sql-wj commented on issue #18698: URL: https://github.com/apache/dolphinscheduler/issues/18698#issuecomment-6143012210
Describe the bug After #18406 (fixed by #18415 in 3.4.3), an architectural gap remains: the "persist task failure to DB" step and the "advance the workflow" step are two non-atomic operations. If the master crashes / restarts / fails over between them, the DB is left with `task = FAILURE` while `workflow = RUNNING`. The serial lock (`t_ds_serial_command`) is not released, so subsequent serial schedules of the same workflow can never trigger. #18406 only fixes one specific DAG shape (Continue failure strategy + a blocked intermediate predecessor). It does not provide any independent reconciliation for the serial lock. ## Which version of DolphinScheduler 3.4.2 (observed); 3.4.3 partially fixed (#18406/#18415). The non-atomicity and the missing serial-lock reconciliation are NOT fixed in 3.4.3. ## To Reproduce 1. Use a workflow whose task can fail. 2. Make the master crash/restart/failover in the window between the task-failure write to DB and the workflow-advance steps (or otherwise get the workflow stuck while a task is FAILURE). 3. Observe: `t_ds_task_instance.state = FAILURE`, `t_ds_workflow_instance.state = RUNNING_EXECUTION`, and a stale row remains in `t_ds_serial_command`. 4. Subsequent serial schedules of the same workflow never trigger. ## Root cause In `AbstractTaskStateAction.onFailedEvent()`: ```java releaseTaskInstanceResourcesIfNeeded(taskExecution); persistentTaskInstanceFailedEventToDB(taskExecution, ...); // (1) persisted to DB ... graph.markTaskExecutionChainFailure(taskExecution); // (2) in-memory only publishWorkflowInstanceTopologyLogicalTransitionEvent(...); // (3) in-memory only ``` `persistentTaskInstanceFailedEventToDB` is a bare `taskInstanceDao.updateById(...)` with no `@Transactional`. Steps (2) and (3) live only in master memory. If the master dies after (1) but before (2)/(3) complete, steps (2)/(3) vanish with the memory, leaving `task = FAILURE, workflow = RUNNING` in the DB. On failover, the graph is rebuilt from the DB, but the failed task is not reliably re-marked into a failure chain and the workflow terminal event is not triggered — `emitWorkflowFinishedEventIfApplicable` returns early when `!isWorkflowFinishable`. The workflow stays RUNNING. Additionally, `WorkflowSerialCoordinator.doStart()` (lines 93–109) only handles `SERIAL_WAIT` / `DISCARD` / `PRIORITY`. There is no logic to scan terminal/zombie workflows and recycle their stale `t_ds_serial_command` rows. Serial-lock cleanup depends 100% on `workflowFinish` being reached. ## Proposed improvement Add a reconciliation task to `WorkflowSerialCoordinator` that scans already-terminal / zombie workflows and recycles their residual `t_ds_serial_command` rows. This is complementary to the #18406 fix (which only covers the Continue + blocked-predecessor shape). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
