Naman-555 commented on issue #16016:
URL: https://github.com/apache/iceberg/issues/16016#issuecomment-6817822976

   Another confirmation of this leak. In our case the trigger was an S3 write 
failure while the task's writers were being closed (`SinkWriter.close()` → 
"Failed to close partitioned delta writer"); the write failure itself was a 
permissions issue on our side. The task was killed, but its coordinator thread 
kept running in the Connect worker JVM with the old catalog, attempting a 
commit every cycle on its own schedule regardless of task restarts. Only 
restarting the worker clears it.
   
   #18909 fixes the leak, not the write failure: `CommitterImpl.close()` now 
stops the coordinator even when stopping the worker throws, and still rethrows 
the original error. Any failure in `stopWorker()` (network, storage, 
permissions) no longer leaves a coordinator behind.
   
   Relation to #18039: fencing ends a leaked coordinator once a new coordinator 
starts and the old one makes its next transactional write. It doesn't prevent 
the leak, so if the connector stays failed, is paused, or is deleted, the 
thread and its Kafka clients and catalog keep running until the worker 
restarts. The two changes are complementary.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to