[
https://issues.apache.org/jira/browse/HADOOP-15392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16452034#comment-16452034
]
Voyta commented on HADOOP-15392:
--------------------------------
[~mackrorysd] It’s not caused by MapReduce job, but by hbase ExportSnapshot
utility.
Here is our finding so far:
* We call the following command:
hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot
-Dfs.s3a.buffer.dir=./s3a-buffer-dir -snapshot <snapshot_name> -copy-to
s3a://<S3_bucket_name>/<prefix>/ -copy-from
hdfs://<hdfs_server_hostname>:8020/hbase -chuser hbase -chgroup hbase
-bandwidth 20
* We have around 50,000 files exported per snapshot
* org.apache.hadoop.hbase.snapshot.ExportSnapshot class uses FileSystem
instances that are S3AFileSystem instances in our case
* An S3AFileSystem instance calls in initialization phase
S3AInstrumentation#registerAsMetricsSource() that creates new unique metric
name (because of counter _metricsSourceActiveCounter_)
* An S3AInstrumentation instance is registered in MetricsSystemImpl by calling
MetricsSystemImpl#registerSource(String name, String desc, MetricsSource
source)
* The S3AInstrumentation instance is the source argument
* There is a single instance org.apache.hadoop.metrics2.impl.MetricsSystemImpl
instantiated by org.apache.hadoop.fs.s3a.S3AInstrumentation#getMetricsSystem()
and the MetricsSystemImpl instance holds all references in
private final Map<String, MetricsSourceAdapter> sources
So far, we don’t see any place where we can switch this off or avoid the
accumulation by a configurable option. It might, however, be related to
https://issues.apache.org/jira/browse/HBASE-20433
Our observation for 1 GB max heap setting after out of memory heap dump was
obtained is the following:
* approx. 53,000 S3AInstrumentation instances and its referenced instances
(e.g. a MetricRegistry instance)
* approx. 53,000 long[] instances referenced by SampleQuantiles (216 MB size)
* approx. 2,700,000 org.apache.hadoop.metrics2.lib.MutableCounterLong
instances (92 MB size)
* approx. 2,700,000 javax.management.MBeanAttributeInfo instances (140 MB
size)
* approx. 2,700,000 javax.management.Attribute instances (87 MB)
* approx. 4,000,000 HashMap$Entry instances (170 MB)
* majority of memory is occupied by metric-related instances (approx. 90%)
> S3A Metrics in S3AInstrumentation Cause Memory Leaks
> ----------------------------------------------------
>
> Key: HADOOP-15392
> URL: https://issues.apache.org/jira/browse/HADOOP-15392
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.1.0
> Reporter: Voyta
> Priority: Major
>
> While using HBase S3A Export Snapshot utility we started to experience memory
> leaks of the process after version upgrade.
> By running code analysis we traced the cause to revision
> 6555af81a26b0b72ec3bee7034e01f5bd84b1564 that added the following static
> reference (singleton):
> private static MetricsSystem metricsSystem = null;
> When application uses S3AFileSystem instance that is not closed immediately
> metrics are accumulated in this instance and memory grows without any limit.
>
> Expectation:
> * It would be nice to have an option to disable metrics completely as this
> is not needed for Export Snapshot utility.
> * Usage of S3AFileSystem should not contain any static object that can grow
> indefinitely.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
