abmo-x commented on PR #5311: URL: https://github.com/apache/iceberg/pull/5311#issuecomment-1190837724
@rdblue @RussellSpitzer Added a commit to clear currentWriter on close in BaseTaskWriter and added 2 test cases around failure to close and complete. I agree close should be only called once and we are relying on that behavior quite strongly and adding the data files. However I have found the writers are held and closed more than once in various scenarios which causes this issue where a close resulted in failure and writers were in a bad state. 1. when user defined functions catch all exceptions and ignore failures on write as seen in Flink's processElement which internally triggers a roll to new file. 2. This behavior was also observed before and fix was made in https://github.com/apache/iceberg/pull/1749 Let me know your thoughts. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
