Mate Czagany created FLINK-40595:
------------------------------------
Summary: Native S3 always aborts multipart uploads required for
checkpoint recovery
Key: FLINK-40595
URL: https://issues.apache.org/jira/browse/FLINK-40595
Project: Flink
Issue Type: Bug
Components: FileSystems
Reporter: Mate Czagany
`NativeS3RecoverableFsDataOutputStream.close()` unconditionally aborts its
multipart upload, including after `persist()` has returned recoverable state
referencing that upload.
This method is called during task shutdown or cancellation to dispose an
in-progress part file. Restoring from a checkpoint will re-use the aborted
upload ID, leading to a failure of `NoSuchUpload`.
My suggestion is to not abort the MPUs, and accept that there will be leaked
MPUs, just like it is with the Hadoop connector right now. These uploads will
require external cleanup, such as an S3 lifecycle policy.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
