[
https://issues.apache.org/jira/browse/FLINK-25023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17498836#comment-17498836
]
David Morávek commented on FLINK-25023:
---------------------------------------
We had further discussions with [~chesnay] on the issue and we don't feel that
the complexity of the fix out-weights the issue.
In worst case scenario, this will leak a single class-loader (of the first job
that has accessed HDFS), which means that there is a constant overhead that can
be count for.
> we don't leak the class loader, we still have the
> StatisticsDataReferenceCleaner thread in taskmanager thread dump and
> metaspace usage isn't decreasing after job finishes
This depends on when the class references get garbage collected. This highly
depends on the GC and current memory usage of the Java heap. Of course you
won't be able to unload hadoop classes as these are still needed by the
reference cleaner -> which is by definition a constant overhead leak by itself
and we have no way of closing this after the job finishes.
> ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of
> user code
> ------------------------------------------------------------------------------------
>
> Key: FLINK-25023
> URL: https://issues.apache.org/jira/browse/FLINK-25023
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hadoop
> Compatibility, FileSystems
> Affects Versions: 1.14.0, 1.12.5, 1.13.3
> Reporter: Nico Kruber
> Assignee: David Morávek
> Priority: Major
> Labels: pull-request-available, stale-assigned
>
> If a Flink job is using HDFS through Flink's filesystem abstraction (either
> on the JM or TM), that code may actually spawn a few threads, e.g. from
> static class members:
> *
> {{org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner}}
> * {{IPC Parameter Sending Thread#*}}
> These threads are started as soon as the classes are loaded which may be in
> the context of the user code. In this specific scenario, however, the created
> threads may contain references to the context class loader (I did not see
> that though) or, as happened here, it may inherit thread contexts such as the
> {{ProtectionDomain}} (from an {{{}AccessController{}}}).
> Hence user contexts and user class loaders are leaked into long-running
> threads that are run in Flink's (parent) classloader.
> Fortunately, it seems to only *leak a single* {{ChildFirstClassLoader}} in
> this concrete example but that may depend on which code paths each client
> execution is walking.
>
> A *proper solution* doesn't seem so simple:
> * We could try to proactively initialize available file systems in the hope
> to start all threads in the parent classloader with parent context.
> * We could create a default {{ProtectionDomain}} for spawned threads as
> discussed at [https://dzone.com/articles/javalangoutofmemory-permgen],
> however, the {{StatisticsDataReferenceCleaner}} isn't actually actively
> spawned from any callback but as a static variable and this with the class
> loading itself (but maybe this is still possible somehow).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
