zhangshenghang opened a new issue, #13325:
URL: https://github.com/apache/seatunnel/issues/13325

   ### What happened
   
   The per-vertex cancel retry loop has no attempt cap and runs while holding 
two levels of monitors, so a live-but-unresponsive worker can block the 
pipeline's entire state machine — including a user's cancel request (dev @ 
89f91a89af):
   
   1. `PhysicalVertex.noticeTaskExecutionServiceCancel` 
(`PhysicalVertex.java:457-507`) retries `CancelTaskOperation` **forever** while 
the target member is alive (`while (!taskFuture.isDone() && executionAddress != 
null && member != null)`, `Thread.sleep(2000)` between failed attempts); `i` is 
only used for logging.
   2. It runs inside `cancel()` → `stateProcess` (both `synchronized` on the 
PhysicalVertex), reached synchronously from `SubPlan.updatePipelineState → 
stateProcess` (both `synchronized` on the SubPlan, `SubPlan.java:421-473`).
   
   So while a worker is up but not answering (long GC, severe overload — each 
`invoke().get()` taking up to the invocation timeout), the thread holds the 
SubPlan monitor: every `updatePipelineState`, `cancelPipeline`, end-state 
callback and restore step for that pipeline queues behind it. A user cancelling 
the job (`PhysicalPlan.cancelJob` → `cancelPipeline`) cannot make progress 
until every invoke either succeeds or the member is declared gone.
   
   ### Impact
   
   Pipeline state machine (including cancellation) frozen for minutes; in 
pathological cases the job cannot be cancelled without killing the worker or 
master.
   
   ### Suggested fix
   
   Bound the retry attempts (then mark the vertex CANCELED locally, as the 
member-unavailable branch at `PhysicalVertex.java:500-506` already does), 
and/or move the cancel sends off the state-machine thread (async with a 
completion callback).
   
   ### SeaTunnel Version
   
   dev (89f91a89af), Zeta engine
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to