zhangshenghang opened a new issue, #13317:
URL: https://github.com/apache/seatunnel/issues/13317

   ### What happened
   
   Terminal task-status events carry no attempt/execution identifier, while the 
delivery mechanism *guarantees* they can arrive after a restore. A late 
terminal event from the previous attempt then overwrites the new attempt's 
state and releases the new attempt's slot.
   
   Chain (dev @ 89f91a89af):
   
   1. Every deployment gets a fresh executionId 
(`PhysicalVertex.getTaskGroupImmutableInformation`, 
`PhysicalVertex.java:339-353`, `flakeIdGenerator.newId()`), but 
`NotifyTaskStatusOperation` only carries `TaskGroupLocation` — the master 
cannot tell attempts apart.
   2. `TaskExecutionService.notifyTaskStatusToMaster` retries terminal-status 
delivery **indefinitely**, and specifically retries through 
`JobRestoreInProgressException` until the master finishes restoring 
(`TaskExecutionService.java:760-815`). So an old attempt's terminal report is 
mechanically guaranteed to be deliverable *after* the restore completes.
   3. On arrival, `JobMaster.updateTaskExecutionState` → 
`PhysicalVertex.updateStateByExecutionService` (`PhysicalVertex.java:575-584`) 
→ `updateTaskState`: the end-state guard (`PhysicalVertex.java:410-413`) no 
longer protects, because the restore's `reset()` already put the vertex back to 
CREATED.
   4. Consequences: a late FAILED/CANCELED fails the healthy new attempt 
(another cancel + restore round, burning retry budget); a late FINISHED 
corrupts the end-state accounting; and `JobMaster.updateTaskExecutionState` 
releases the task-group resource for **any** terminal event 
(`JobMaster.java:1383-1389`) — releasing the slot the *new* attempt just 
obtained, while its task keeps running.
   
   ### Trigger sequence
   
   Worker flaps (memberRemoved) → its tasks marked FAILED → pipeline restores 
and redeploys → worker rejoins → the old task truly ends and its (retried) 
terminal report lands on the new attempt.
   
   ### Impact
   
   Healthy restored pipelines killed by ghost events; retry budget consumed; 
running tasks with released slots (slot double-assignment risk).
   
   ### Suggested fix
   
   Carry the deployment's executionId (or an attempt epoch) on 
`NotifyTaskStatusOperation`/`TaskExecutionState` and drop events whose id does 
not match the current deployment; validate slot ownership (sequence) before 
releasing on a terminal event.
   
   ### SeaTunnel Version
   
   dev (89f91a89af), Zeta engine
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to