[
https://issues.apache.org/jira/browse/FLINK-40595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Somogyi reassigned FLINK-40595:
-------------------------------------
Assignee: Mate Czagany
> Native S3 always aborts multipart uploads required for checkpoint recovery
> --------------------------------------------------------------------------
>
> Key: FLINK-40595
> URL: https://issues.apache.org/jira/browse/FLINK-40595
> Project: Flink
> Issue Type: Bug
> Components: FileSystems
> Reporter: Mate Czagany
> Assignee: Mate Czagany
> Priority: Major
> Labels: pull-request-available
>
> `NativeS3RecoverableFsDataOutputStream.close()` unconditionally aborts its
> multipart upload, including after `persist()` has returned recoverable state
> referencing that upload.
> This method is called during task shutdown or cancellation to dispose an
> in-progress part file. Restoring from a checkpoint will re-use the aborted
> upload ID, leading to a failure of `NoSuchUpload`.
> My suggestion is to not abort the MPUs, and accept that there will be leaked
> MPUs, just like it is with the Hadoop connector right now. These uploads will
> require external cleanup, such as an S3 lifecycle policy.
> The promise of FLIP-555 to "ensure that orphaned multipart uploads are
> cleaned up" is not really true now anyways, and I don't see any reliable way
> to make it happen. For now I think we should fix this correctness gap, and
> then look into how to properly clean up the orphaned MPUs in a separate ticket
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
