[
https://issues.apache.org/jira/browse/HADOOP-15392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16452659#comment-16452659
]
Steve Loughran commented on HADOOP-15392:
-----------------------------------------
OK< it's in the launcher? And there are 53K instances of S3A FS there? This
sounds like a new instance is being created every time rather than the cache
being involved.
There's more at stake than just metrics: each s3a instance creates a thread
pool, which is v. expensive if the runtime starts allocating memory for each
thread's stack, plus a pool of HTTP connections, which, if kept open, will use
up a lot of TCP connections. What does netstat -a say on this machine?
Alternatively, if it's just the S3AInstrumentations which are hanging around,
maybe there's some loop of ref counting going on?
I think we ought to look @ the hbase problem independently of the memory
management of the metrics.
Voyta: thanks for your research here, it really helps us understand what's up.
One thing, do make sure that fs.s3a.impl.disable.cache is either unset or
false, as that would create lots of S3a instances.
> S3A Metrics in S3AInstrumentation Cause Memory Leaks in HBase Export
> --------------------------------------------------------------------
>
> Key: HADOOP-15392
> URL: https://issues.apache.org/jira/browse/HADOOP-15392
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.1.0
> Reporter: Voyta
> Priority: Blocker
>
> While using HBase S3A Export Snapshot utility we started to experience memory
> leaks of the process after version upgrade.
> By running code analysis we traced the cause to revision
> 6555af81a26b0b72ec3bee7034e01f5bd84b1564 that added the following static
> reference (singleton):
> private static MetricsSystem metricsSystem = null;
> When application uses S3AFileSystem instance that is not closed immediately
> metrics are accumulated in this instance and memory grows without any limit.
>
> Expectation:
> * It would be nice to have an option to disable metrics completely as this
> is not needed for Export Snapshot utility.
> * Usage of S3AFileSystem should not contain any static object that can grow
> indefinitely.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
