[
https://issues.apache.org/jira/browse/HADOOP-15392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16452677#comment-16452677
]
Steve Loughran commented on HADOOP-15392:
-----------------------------------------
And of course, what do we see in ExportSnapshot [line
792|https://github.com/apache/hbase/blob/master/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java#L942]
{code}
srcConf.setBoolean("fs." + inputRoot.toUri().getScheme() +
".impl.disable.cache", true);
{code}
That is: the entry point process does disable fs caching in both src and dest.
Now, it is cleaning them up in [Line
1074|https://github.com/apache/hbase/blob/master/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java#L1074],
but it's got me worried. It'd be better if they used
FileSystem.newInstance(URI, config), so there'd be no altering of configs. Even
so, I don't see from looking at the code where the other 52998 are coming from.
> S3A Metrics in S3AInstrumentation Cause Memory Leaks in HBase Export
> --------------------------------------------------------------------
>
> Key: HADOOP-15392
> URL: https://issues.apache.org/jira/browse/HADOOP-15392
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.1.0
> Reporter: Voyta
> Priority: Blocker
>
> While using HBase S3A Export Snapshot utility we started to experience memory
> leaks of the process after version upgrade.
> By running code analysis we traced the cause to revision
> 6555af81a26b0b72ec3bee7034e01f5bd84b1564 that added the following static
> reference (singleton):
> private static MetricsSystem metricsSystem = null;
> When application uses S3AFileSystem instance that is not closed immediately
> metrics are accumulated in this instance and memory grows without any limit.
>
> Expectation:
> * It would be nice to have an option to disable metrics completely as this
> is not needed for Export Snapshot utility.
> * Usage of S3AFileSystem should not contain any static object that can grow
> indefinitely.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
