[
https://issues.apache.org/jira/browse/HDFS-12748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16233761#comment-16233761
]
Weiwei Yang commented on HDFS-12748:
------------------------------------
Thanks [~daryn], your comment makes sense to me. Just uploaded v2 patch, this
patch pulls some common methods out for re-use, and remove the FileSystem call
for GETHOMEDIRECTORY, please help to review, thanks.
Note, GETTRASHROOT has same issue, but it requires more refactor (related to
EC) to make it work consistent in webhdfs and HDFS, I think we need a separate
JIRA to fix.
Please let me know if this makes sense, thanks.
> NameNode memory leak when accessing webhdfs GETHOMEDIRECTORY
> ------------------------------------------------------------
>
> Key: HDFS-12748
> URL: https://issues.apache.org/jira/browse/HDFS-12748
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
> Affects Versions: 2.8.2
> Reporter: Jiandan Yang
> Assignee: Weiwei Yang
> Priority: Major
> Attachments: HDFS-12748.001.patch, HDFS-12748.002.patch
>
>
> In our production environment, the standby NN often do fullgc, through mat we
> found the largest object is FileSystem$Cache, which contains 7,844,890
> DistributedFileSystem.
> By view hierarchy of method FileSystem.get() , I found only
> NamenodeWebHdfsMethods#get call FileSystem.get(). I don't know why creating
> different DistributedFileSystem every time instead of get a FileSystem from
> cache.
> {code:java}
> case GETHOMEDIRECTORY: {
> final String js = JsonUtil.toJsonString("Path",
> FileSystem.get(conf != null ? conf : new Configuration())
> .getHomeDirectory().toUri().getPath());
> return Response.ok(js).type(MediaType.APPLICATION_JSON).build();
> }
> {code}
> When we close FileSystem when GETHOMEDIRECTORY, NN don't do fullgc.
> {code:java}
> case GETHOMEDIRECTORY: {
> FileSystem fs = null;
> try {
> fs = FileSystem.get(conf != null ? conf : new Configuration());
> final String js = JsonUtil.toJsonString("Path",
> fs.getHomeDirectory().toUri().getPath());
> return Response.ok(js).type(MediaType.APPLICATION_JSON).build();
> } finally {
> if (fs != null) {
> fs.close();
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
