Naman-555 commented on issue #16016: URL: https://github.com/apache/iceberg/issues/16016#issuecomment-6817822976
Another confirmation of this leak. In our case the trigger was an S3 write failure while the task's writers were being closed (`SinkWriter.close()` → "Failed to close partitioned delta writer"); the write failure itself was a permissions issue on our side. The task was killed, but its coordinator thread kept running in the Connect worker JVM with the old catalog, attempting a commit every cycle on its own schedule regardless of task restarts. Only restarting the worker clears it. #18909 fixes the leak, not the write failure: `CommitterImpl.close()` now stops the coordinator even when stopping the worker throws, and still rethrows the original error. Any failure in `stopWorker()` (network, storage, permissions) no longer leaves a coordinator behind. Relation to #18039: fencing ends a leaked coordinator once a new coordinator starts and the old one makes its next transactional write. It doesn't prevent the leak, so if the connector stays failed, is paused, or is deleted, the thread and its Kafka clients and catalog keep running until the worker restarts. The two changes are complementary. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
