Wei-Chiu Chuang created HADOOP-13971:
----------------------------------------
Summary: Fix memory leak in FileSystem.Cache.Key class
Key: HADOOP-13971
URL: https://issues.apache.org/jira/browse/HADOOP-13971
Project: Hadoop Common
Issue Type: Bug
Components: fs, security
Reporter: Wei-Chiu Chuang
Calling {{FileSystem#get(final URI uri, final Configuration conf, final String
user)}} multiple times can result in memory leak because of the hash method
implementation of UserGroupInformation. FileSystem always instantiates a new
FileSystem object despite using the same user name/same URI.
In the past, other downstream projects work around this bug by either disabling
cache (set {{fs.%s.impl.disable.cache}} to {{true}}) or call
{{FileSystem.closeAllForUGI()}} to release resource on demand. (See for
instance HIVE-3098, YARN-58, TEZ-1585)
However, neither approach is desirable. The first workaround loses performance
because it disables cache. This bug was discussed extensively in HADOOP-12707,
but the proposed workaround {{FileSystem.closeAllForUGI()}} is insufficient,
because it won't purge the objects from cache due to the same hash method
implementation bug.
I would like to file a new jira, knowing that current workarounds do not work,
and invite more discussion. An ideal approach is to change UGI hash method, but
it may break many downstream applications, so setting target version as
3.0.0-beta
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
