[
https://issues.apache.org/jira/browse/HADOOP-15392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16452661#comment-16452661
]
Sean Mackrory commented on HADOOP-15392:
----------------------------------------
{quote}I found contain only commented lines{quote}
Okay - thanks for checking. I believe the default is all commented out, except
*.period=10, but I wasn't seeing the MutableQuantiles thread or the metrics
system thread start up with only that either.
The 53,000 instances is still weird. Is fs.s3a.impl.disable.cache set to true?
It's not the default, so if you're not setting it, this shouldn't be happening.
Within a JVM, communication with 1 S3 bucket should usually be done with a
single, cached instance of S3AFileSystem, which should only yield a single
S3AInstrumentation instance and thus the memory usage should 1/53,000th of what
you're seeing.
> S3A Metrics in S3AInstrumentation Cause Memory Leaks in HBase Export
> --------------------------------------------------------------------
>
> Key: HADOOP-15392
> URL: https://issues.apache.org/jira/browse/HADOOP-15392
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.1.0
> Reporter: Voyta
> Priority: Blocker
>
> While using HBase S3A Export Snapshot utility we started to experience memory
> leaks of the process after version upgrade.
> By running code analysis we traced the cause to revision
> 6555af81a26b0b72ec3bee7034e01f5bd84b1564 that added the following static
> reference (singleton):
> private static MetricsSystem metricsSystem = null;
> When application uses S3AFileSystem instance that is not closed immediately
> metrics are accumulated in this instance and memory grows without any limit.
>
> Expectation:
> * It would be nice to have an option to disable metrics completely as this
> is not needed for Export Snapshot utility.
> * Usage of S3AFileSystem should not contain any static object that can grow
> indefinitely.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
