[
https://issues.apache.org/jira/browse/FLINK-9560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511355#comment-16511355
]
ASF GitHub Bot commented on FLINK-9560:
---------------------------------------
Github user pnowojski commented on a diff in the pull request:
https://github.com/apache/flink/pull/6149#discussion_r195134923
--- Diff: docs/ops/filesystems.md ---
@@ -89,9 +89,9 @@ For example, if the default file system configured as
`fs.default-scheme: hdfs:/
#### Connection limiting
You can limit the total number of connections that a file system can
concurrently open. This is useful when the file system cannot handle a large
number
-of concurrent reads / writes or open connections at the same time.
+of concurrent reads / writes or open connections at the same time. You can
also limit the throughput/bandwidth used to read/write from/to the FileSystem
-For example, very small HDFS clusters with few RPC handlers can sometimes
be overwhelmed by a large Flink job trying to build up many connections during
a checkpoint.
+For example, very small HDFS clusters with few RPC handlers can sometimes
be overwhelmed by a large Flink job trying to build up many connections during
a checkpoint.
--- End diff --
white space at the end?
> RateLimiting for FileSystem
> ---------------------------
>
> Key: FLINK-9560
> URL: https://issues.apache.org/jira/browse/FLINK-9560
> Project: Flink
> Issue Type: Improvement
> Components: FileSystem
> Affects Versions: 1.5.0
> Reporter: Etienne CARRIERE
> Priority: Major
>
> *Pain*: On our system, we see that during checkpoint , all the bandwidth is
> take to send the checkpoint to object storage (s3 in our case)
> *Proposal* : After the creation of some limitation on Filesystem (mostly
> number of connections with the tickets FLINK-8125/FLINK-8198/FLINK-9468), I
> propose to add ratelimiting "per Filesystem" .
> *Proposal of implementation* : Modify LimitedConnectionsFileSystem to add a
> ratelimiter on both Input and OutputStream.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
