[ https://issues.apache.org/jira/browse/FLINK-8125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16264345#comment-16264345 ]
Stephan Ewen commented on FLINK-8125: ------------------------------------- [~srichter] Could do that. - pro: Would avoid yet another layer of wrapping. - con: Not active by default, only for some cases needed. Having it separate keeps that code out of the way in most cases. > Support limiting the number of open FileSystem connections > ---------------------------------------------------------- > > Key: FLINK-8125 > URL: https://issues.apache.org/jira/browse/FLINK-8125 > Project: Flink > Issue Type: Improvement > Components: Core > Reporter: Stephan Ewen > Assignee: Stephan Ewen > Fix For: 1.5.0, 1.4.1 > > > We need a way to limit the number of streams that Flink FileSystems > concurrently open. > For example, for very small HDFS clusters with few RPC handlers, a large > Flink job trying to build up many connections during a checkpoint causes > failures due to rejected connections. > I propose to add a file system that can wrap another existing file system The > file system may track the progress of streams and close streams that have > been inactive for too long, to avoid locked streams of taking up the complete > pool. -- This message was sent by Atlassian JIRA (v6.4.14#64029)