[
https://issues.apache.org/jira/browse/FLINK-9560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525534#comment-16525534
]
ASF GitHub Bot commented on FLINK-9560:
---------------------------------------
Github user etiennecarriere commented on the issue:
https://github.com/apache/flink/pull/6149
Hi @pnowojski,
Unfortunately, this patch is not working because :
* The implementation of FileSystem is using a temporary file to store it
** s3 : it is mandatory to know the file of the file before sending it
(Content-Length is needed and Chunked mode is not supported)
https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/model/PutObjectRequest.html
*** flink-s3-fs-hadoop :
https://github.com/Aloisius/hadoop-s3a/blob/master/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L808
*** flink-s3-fs-presto :
https://github.com/prestodb/presto/blob/master/presto-hive/src/main/java/com/facebook/presto/hive/s3/PrestoS3FileSystem.java#L991
* Swift : even if possible to upload file via chunked the hadoop
implementation use tempFile
** flink-swift-fs-hadoop :
https://github.com/c9n/hadoop/blob/master/hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeOutputStream.java#L78
so I propose to close the PR as it is not the right level to handle that.
> RateLimiting for FileSystem
> ---------------------------
>
> Key: FLINK-9560
> URL: https://issues.apache.org/jira/browse/FLINK-9560
> Project: Flink
> Issue Type: Improvement
> Components: FileSystem
> Affects Versions: 1.5.0
> Reporter: Etienne CARRIERE
> Assignee: Etienne CARRIERE
> Priority: Major
> Labels: pull-request-available
>
> *Pain*: On our system, we see that during checkpoint , all the bandwidth is
> take to send the checkpoint to object storage (s3 in our case)
> *Proposal* : After the creation of some limitation on Filesystem (mostly
> number of connections with the tickets FLINK-8125/FLINK-8198/FLINK-9468), I
> propose to add ratelimiting "per Filesystem" .
> *Proposal of implementation* : Modify LimitedConnectionsFileSystem to add a
> ratelimiter on both Input and OutputStream.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
