[
https://issues.apache.org/jira/browse/FLINK-9626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518096#comment-16518096
]
Stefan Richter commented on FLINK-9626:
---------------------------------------
I think in general it can be dangerous to call a {{close}} on filesystems if
the cache is cleared because some client might still reference and use the
instance of the filesystem. Options could be that clients also need to
{{close}} filesystems and we do a reference counting that has to hit zero
before the instance is actually closed internally. Or we could use a safety net
based on {{PhantomReference}}, similar to what we do with guarding input/output
streams. Or a combination of both. Might also be good to hear [~StephanEwen]
about this.
> Possible resource leak in FileSystem
> ------------------------------------
>
> Key: FLINK-9626
> URL: https://issues.apache.org/jira/browse/FLINK-9626
> Project: Flink
> Issue Type: Bug
> Components: FileSystem
> Affects Versions: 1.5.0
> Reporter: Piotr Nowojski
> Priority: Critical
>
> There is a potential resource leak in
> org.apache.flink.core.fs.FileSystem#getUnguardedFileSystem.
> Inside it there is a code:
>
> {code:java}
> // this "default" initialization makes sure that the FileSystem class works
> // even when not configured with an explicit Flink configuration, like on
> // JobManager or TaskManager setup
> if (FS_FACTORIES.isEmpty()) {
> initialize(new Configuration());
> }
> {code}
> which is executed on each cache miss. However this initialize method is also
> doing
>
>
> {code:java}
> CACHE.clear();
> {code}
> without closing file systems in CACHE (this could be problematic for
> HadoopFileSystem which is a wrapper around org.apache.hadoop.fs.FileSystem
> which is closable).
> Now if for example we are constantly accessing two different file systems
> (file systems are differentiated by combination of [schema and
> authority|https://en.wikipedia.org/wiki/Uniform_Resource_Identifier#Generic_syntax]
> part from the file system's URI) initialized from FALLBACK_FACTORY, each
> time we call getUnguardedFileSystem for one of them, that call will clear
> from CACHE entry for the other one. Thus we will constantly be creating new
> FileSystems without closing them.
> Solution could be to either not clear the CACHE or make sure that FileSystems
> are properly closed.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
