jjj-n opened a new issue, #12413:
URL: https://github.com/apache/seatunnel/issues/12413

   ### Search before asking
   
   - [x] I searched existing issues. The closest reports are #8168 and #7339; 
both were closed as stale without a fix.
   
   ### What happened
   
   When a task group is canceled, its `BlockingWorker` can retain or receive an 
interrupt while it is finishing. The last task calls 
`TaskExecutionService.TaskGroupExecutionTracker#taskDone()`, which 
synchronously invokes `updateMetricsContextInImap()` on that interrupted worker 
before completing the task-group future.
   
   If the final metrics write reaches Hazelcast while the thread is 
interrupted, the socket write can throw `ClosedByInterruptException`. Java NIO 
closes the channel in that case, so a best-effort final metrics update can 
close the worker's shared connection to a live master. Because this update runs 
before the terminal future is completed, cancellation can also remain stuck or 
time out.
   
   The current `dev` branch still has the synchronous call in `taskDone()`. 
Periodic metrics updates already run on a separate scheduled executor, but the 
final update does not.
   
   Issue #8168 contains the same production sequence: a canceled 
`BlockingWorker` enters `taskDone()`, `updateMetricsContextInImap()` throws 
`ClosedByInterruptException`, and the shared Hazelcast connection closes. Issue 
#7339 reports the same interrupted-Hazelcast-write mechanism from a checkpoint 
path. Both issues were closed by the stale bot.
   
   ### How to reproduce
   
   A deterministic two-member test can reproduce the failure without mocking 
Hazelcast networking:
   
   1. Start a master and worker and capture their existing Hazelcast connection.
   2. Run a controllable task on the worker.
   3. Cancel the task and inject a late interrupt into the original 
`BlockingWorker` when final metrics collection begins.
   4. Wait for the task group to reach a terminal state and inspect the 
captured connection.
   
   With the current implementation, the negative control throws 
`ClosedByInterruptException`, the captured connection closes, and cancellation 
exceeds a 10-second terminal-state deadline. Moving only the final metrics 
update to the existing task executor while preserving the wait-before-terminal 
ordering makes the test pass and keeps the original connection alive.
   
   ### Expected behavior
   
   Final metrics reporting should not perform Hazelcast I/O on an interrupted 
task worker. The task group should still wait for the final metrics attempt 
before publishing its terminal state, and metrics failures should remain 
best-effort without closing the worker-master connection.
   
   ### Impact
   
   - A normal cancellation path can close a healthy, shared Hazelcast member 
connection.
   - The final metrics snapshot can be lost.
   - Task-group cancellation can be delayed or remain in `CANCELING`.
   - Other operations sharing the connection can be disrupted until 
reconnection.
   
   ### Scope
   
   This issue tracks only the final metrics call from 
`TaskExecutionService#taskDone()`. It does not claim to fix every interrupted 
Hazelcast write path or the separate worker-loss `CANCELED`/`FAILED` race.
   
   I have a focused candidate patch with one unit regression test and one 
two-member regression test and can submit it as a separate PR.
   
   ### SeaTunnel version
   
   Current `dev` (`3.0.0-SNAPSHOT`); the same stack was reported against 2.3.8 
in #8168.
   
   ### Java version
   
   Reproduced with Java 8. The underlying `ClosedByInterruptException` behavior 
is part of Java NIO channel interruption semantics.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to