[ 
https://issues.apache.org/jira/browse/FLINK-8125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16264345#comment-16264345
 ] 

Stephan Ewen commented on FLINK-8125:
-------------------------------------

[~srichter] Could do that.

  - pro: Would avoid yet another layer of wrapping.
  - con: Not active by default, only for some cases needed. Having it separate 
keeps that code out of the way in most cases.

> Support limiting the number of open FileSystem connections
> ----------------------------------------------------------
>
>                 Key: FLINK-8125
>                 URL: https://issues.apache.org/jira/browse/FLINK-8125
>             Project: Flink
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Stephan Ewen
>            Assignee: Stephan Ewen
>             Fix For: 1.5.0, 1.4.1
>
>
> We need a way to limit the number of streams that Flink FileSystems 
> concurrently open.
> For example, for very small HDFS clusters with few RPC handlers, a large 
> Flink job trying to build up many connections during a checkpoint causes 
> failures due to rejected connections. 
> I propose to add a file system that can wrap another existing file system The 
> file system may track the progress of streams and close streams that have 
> been inactive for too long, to avoid locked streams of taking up the complete 
> pool.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Reply via email to