[
https://issues.apache.org/jira/browse/FLINK-24392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser updated FLINK-24392:
-----------------------------------
Description:
The Presto s3 filesystem implementation currently shipped with Flink doesn't
support streaming uploads. All data needs to be materialized to a single file
on disk, before it can be uploaded.
This can lead to situations where TaskManagers are running out of disk when
creating a savepoint.
The Hadoop filesystem implementation supports streaming uploads (by using
multipart uploads of smaller (say 100mb) files locally), but it does more API
calls, leading to other issues.
Trino 348 supports streaming uploads.
was:
The Presto s3 filesystem implementation currently shipped with Flink doesn't
support streaming uploads. All data needs to be materialized to a single file
on disk, before it can be uploaded.
This can lead to situations where TaskManagers are running out of disk when
creating a savepoint.
The Hadoop filesystem implementation supports streaming uploads (by using
multipart uploads of smaller (say 100mb) files locally), but it does more API
calls, leading to other issues.
Trinion 348 supports streaming uploads.
> Upgrade presto s3 fs implementation to Trino >= 348
> ---------------------------------------------------
>
> Key: FLINK-24392
> URL: https://issues.apache.org/jira/browse/FLINK-24392
> Project: Flink
> Issue Type: Improvement
> Components: FileSystems
> Affects Versions: 1.14.0
> Reporter: Robert Metzger
> Priority: Major
> Fix For: 1.15.0
>
>
> The Presto s3 filesystem implementation currently shipped with Flink doesn't
> support streaming uploads. All data needs to be materialized to a single file
> on disk, before it can be uploaded.
> This can lead to situations where TaskManagers are running out of disk when
> creating a savepoint.
> The Hadoop filesystem implementation supports streaming uploads (by using
> multipart uploads of smaller (say 100mb) files locally), but it does more API
> calls, leading to other issues.
> Trino 348 supports streaming uploads.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
