Robert Metzger created FLINK-24392:
--------------------------------------
Summary: Upgrade presto s3 fs implementation to Trinio >= 348
Key: FLINK-24392
URL: https://issues.apache.org/jira/browse/FLINK-24392
Project: Flink
Issue Type: Improvement
Components: FileSystems
Affects Versions: 1.14.0
Reporter: Robert Metzger
Fix For: 1.15.0
The Presto s3 filesystem implementation currently shipped with Flink doesn't
support streaming uploads. All data needs to be materialized to a single file
on disk, before it can be uploaded.
This can lead to situations where TaskManagers are running out of disk when
creating a savepoint.
The Hadoop filesystem implementation supports streaming uploads (by using
multipart uploads of smaller (say 100mb) files locally), but it does more API
calls, leading to other issues.
Trinion 348 supports streaming uploads.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
