zhangshenghang opened a new issue, #13315: URL: https://github.com/apache/seatunnel/issues/13315
Related to #12314 (same fire-and-forget reporting family, different operation and failure mode). ### What happened If a task's `restoreState` fails and the error report to the master is lost, the task spins in WAITING_RESTORE forever and nothing ever times the phase out: the pipeline stays RUNNING with no progress and no error. Chain (dev @ 89f91a89af): 1. Task side: `SeaTunnelTask.stateProcess` spins in `WAITING_RESTORE` every 100 ms until `restoreComplete` is done (`SeaTunnelTask.java:267-276`). `restoreComplete` only completes inside `restoreState` on success. 2. Worker side: `NotifyTaskRestoreOperation` runs `task.restoreState(...)` asynchronously and, on failure, only does `sendToMaster(new CheckpointErrorReportOperation(...))` — **no join, no retry** (`NotifyTaskRestoreOperation.java:111-115`). 3. Master side: `CheckpointCoordinator.restoreTaskState` only `.join()`s the operation *delivery* (`CheckpointCoordinator.java:430-433`); `runInternal` returns right after submitting the async function, so delivery success says nothing about restore success. 4. If the report is lost (invocation failure during a partition or master switchover), `allTaskReady` never satisfies the size check (`CheckpointCoordinator.java:459-461`), and **no scheduler exists before `allTaskReady`** (`CheckpointCoordinator.java:476-478`) — there is no timeout covering this phase at all (checkpoint timeout only guards already-triggered pending checkpoints). Contrast: terminal task-status reporting retries indefinitely (`TaskExecutionService.notifyTaskStatusToMaster`, incl. a dedicated `JobRestoreInProgressException` retry path), so the two reporting channels have deliberately different reliability — and the restore phase uses the unreliable one. ### Impact Pipeline RUNNING forever with zero indication; only workaround is manual restart. ### Suggested fix - Retry the `CheckpointErrorReportOperation` the way `notifyTaskStatusToMaster` retries terminal states, and/or - Add a configurable timeout for the WAITING_RESTORE phase (restore start → READY_START per task) that fails the task and triggers the normal pipeline recovery. ### SeaTunnel Version dev (89f91a89af), Zeta engine -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
