[ https://issues.apache.org/jira/browse/HADOOP-15392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16442784#comment-16442784 ]
Steve Loughran commented on HADOOP-15392: ----------------------------------------- That Export Snapshot gets filesystems in two places * [Mapper|https://github.com/apache/hbase/blob/master/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java#L224]: created in setup, destroyed in cleanup. Life should be that of the mapper alone. * CLI entry point [doWork|https://github.com/apache/hbase/blob/master/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java#L915]. Again, close() at the end, which is more than I often no. I don't see any leaking of FS instances anywhere: nothing in the HBase code to point a finger at. If we are leaking memory in the metrics is because the thing uses lots of memory *and* the ref counting isn't working. # Is there a config option we can set up to not accrue stuff by default? # maybe it's the fact that this is running with HBase configs which is making it surface. @Krizek: what is your metrics config file here? # If there's no way to stop this happening them the metric wire-up has to be optional; keep off by default. > S3A Metrics in S3AInstrumentation Cause Memory Leaks > ---------------------------------------------------- > > Key: HADOOP-15392 > URL: https://issues.apache.org/jira/browse/HADOOP-15392 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 > Affects Versions: 3.1.0 > Reporter: Voyta > Priority: Major > > While using HBase S3A Export Snapshot utility we started to experience memory > leaks of the process after version upgrade. > By running code analysis we traced the cause to revision > 6555af81a26b0b72ec3bee7034e01f5bd84b1564 that added the following static > reference (singleton): > private static MetricsSystem metricsSystem = null; > When application uses S3AFileSystem instance that is not closed immediately > metrics are accumulated in this instance and memory grows without any limit. > > Expectation: > * It would be nice to have an option to disable metrics completely as this > is not needed for Export Snapshot utility. > * Usage of S3AFileSystem should not contain any static object that can grow > indefinitely. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org