gaborgsomogyi commented on PR #29132: URL: https://github.com/apache/flink/pull/29132#issuecomment-5618399117
Makese sense. Stepping back: I've now found two separate cases where selective abort corrupts recoverable state (`close()` after `persist()`, and `closeForCommit()`-failure after `persist()`). Any such scheme relies on proving "no recoverable references this uploadId" from process-local state, and a power outage alone proves that guarantee is never airtight. A lifecycle policy is mandatory regardless of what the code does. Given that, proactive abort only buys marginally faster cleanup within a retention window that must already be conservative for correctness reasons. That's not worth the recurring correctness risk of a hand-rolled flag/state machine that's already needed patching twice. Proposal: drop abort entirely, in both `close()` and `closeForCommit()`'s failure catch, matching the Hadoop connector. Rely on the documented S3 lifecycle rule for cleanup. @Samrat002 WDYT since you've added that? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
