SEPURI-SAI-KRISHNA opened a new issue, #18540: URL: https://github.com/apache/dolphinscheduler/issues/18540
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues. ### What happened When a workflow is recovered with "Recover failed tasks" (`CommandType.START_FAILURE_TASK_PROCESS`), `RecoverFailureTaskCommandHandler` recreates every `FAILURE` / `KILL` task instance through `FailedRecoverTaskInstanceFactory`. That factory clones the old task instance, clears part of its runtime state and inserts it as a brand-new row, marking the old one `flag = NO`: ```java final TaskInstance taskInstance = cloneTaskInstance(needRecoverTaskInstance); taskInstance.setId(null); taskInstance.setState(TaskExecutionStatus.SUBMITTED_SUCCESS); taskInstance.setHost(null); taskInstance.setVarPool(null); taskInstance.setSubmitTime(new Date()); taskInstance.setLogPath(null); taskInstance.setExecutePath(null); taskInstanceDao.insert(taskInstance); ``` `AbstractTaskInstanceFactory#cloneTaskInstance` copies `retryTimes`, `startTime`, `endTime`, `pid` and `alertFlag`, and the factory never resets them. A task that failed after using up its retries has `retryTimes == maxRetryTimes`. The recreated instance therefore starts life with a **fully consumed retry budget**, because `TaskExecution#isTaskInstanceCanRetry()` is ```java return taskInstance.getRetryTimes() < taskInstance.getMaxRetryTimes(); ``` So on the recovery run the task fails permanently on its very first failure and is never retried, even though the task definition asks for N retries. The stale `startTime` / `endTime` are also carried over, so until the task actually starts the new instance is displayed with the timestamps of the previous failed attempt, and `TaskInstanceServiceImpl#queryTaskListPaging` computes its duration from them (`DateUtils.format2Duration(startTime, endTime)`). ### What you expected to happen The recreated task instance is a new attempt, so it should get the whole retry budget back and should not inherit the runtime state of the failed attempt — exactly like the two sibling factories do: * `FirstRunTaskInstanceFactory` sets `retryTimes(0)`, `startTime(null)`, `endTime(null)`, `alertFlag(NO)`. * `RetryTaskInstanceFactory` sets `startTime(null)`, `endTime(null)`, `pid(0)` and deliberately increments `retryTimes`. `FailedRecoverTaskInstanceFactory` is the odd one out. ### How to reproduce 1. Create a workflow with one task that fails, and set **Failed retry times = 3**, **Failed retry interval = 1**. 2. Run it. The task retries 3 times, then the task and the workflow end in `FAILURE`. The last task instance has `retry_times = 3`, `max_retry_times = 3`. 3. On the workflow instance, click **Recover failed tasks**. 4. Expected: the task runs again and, if it keeps failing, retries up to 3 more times. Actual: the task runs once, fails, and is not retried at all — the workflow goes straight back to `FAILURE`. 5. Check `t_ds_task_instance`: the newly inserted row already has `retry_times = 3`, plus the `start_time` / `end_time` / `pid` of the previous failed attempt. ### Anything else Every recovery of a retry-exhausted task is affected, so the workaround is to click "Recover failed tasks" repeatedly, which is what the retry configuration was supposed to do automatically. This is the same class of bug as #16991 (`FailedRecoverTaskInstanceFactory` not setting `environmentConfig`), in the same factory, but a different set of fields. #16991 was resolved by resolving the environment centrally in `TaskExecutionContextFactory`, so the two do not overlap. ### Version dev ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
