[
https://issues.apache.org/jira/browse/HADOOP-15392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16451243#comment-16451243
]
Sean Mackrory edited comment on HADOOP-15392 at 4/24/18 9:25 PM:
-----------------------------------------------------------------
{quote}if you or someone else would care to provide a switch to turn this
off{quote}
If there's a problem we can't fix I'd be happy to, but there shouldn't be any
significant difference between how the relatively short-lived connector is
accumulating metrics and how long-running daemons do it. If there's a memory
leak then either we're doing it wrong and we should fix it (unless the fix is
worse than disabling metrics by default or something), or this is a very big
problem indeed that may be affecting other Hadoop daemons. I ran a profiler and
saw 2 metrics-related threads pop up: one is an executor for MetricsSinkAdapter
that tends to grow by about 6kb every period of the metrics system (which by
default is 10 seconds) and there's another one for MutableQuantiles that grows
by about 64 bytes per second. Now I don't see why either of those needs to grow
indefinitely like that, but that rate also sounds insignificant compared to the
kind of growth being described here, and the behavior is the same between the
other Hadoop daemons that were unaffected by the change in question and S3a
clients. It also doesn't appear to happen unless you have metrics sinks
configured, etc. and the default configuration doesn't have any of that. I also
ran a bunch of HBase snapshot exports to S3 both with and without this change
and I'm not seeing any particular pattern in the memory usage that matches what
you're describing. So a couple of follow-up questions:
* To be clear, exactly where are you seeing the growth in memory usage? As I
understand it this is in the MapReduce job that exports snapshots to S3, right?
If that's the case, can you identify a particular thread that seems to be
accumulating all the memory? I initially thought you were referring to one of
the HBase daemons, but if this is the MR job then whether or not it closes the
FileSystem is probably rather academic because it would close the FileSystem as
the JVM was about to shut down anyway, so it likely doesn't affect whether or
not you see a problem with memory usage.
* Do you have anything in hadoop-metrics2.properties at all?
was (Author: mackrorysd):
{quote}if you or someone else would care to provide a switch to turn this
off{quote}
If there's a problem we can't fix I'd be happy to, but there shouldn't be any
significant difference between how the relatively short-lived connector is
accumulating metrics and how long-running daemons do it. If there's a memory
leak then either we're doing it wrong and we should fix it (unless the fix is
worse than disabling metrics by default or something), or this is a very big
problem indeed that may be affecting other Hadoop daemons. I ran a profiler and
saw 2 metrics-related threads pop up: one is an executor for MetricsSinkAdapter
that tends to grow by about 6kb every period of the metrics system (which by
default is 10 seconds) and there's another one for MutableQuantiles that grows
by about 64 bytes per second. Now I don't see why either of those needs to grow
indefinitely like that, but that rate also sounds insignificant compared to the
kind of growth being described here, and the behavior is the same between the
other Hadoop daemons that were unaffected by the change in question and S3a
clients. It also doesn't appear to happen unless you have metrics sinks
configured, etc. and the default configuration doesn't have any of that. I also
ran a bunch of HBase snapshot exports to S3 both with and without this change
and I'm not seeing any particular pattern in the memory usage that matches what
you're describing. So a couple of follow-up questions:
*To be clear, exactly where are you seeing the growth in memory usage? As I
understand it this is in the MapReduce job that exports snapshots to S3, right?
If that's the case, can you identify a particular thread that seems to be
accumulating all the memory? I initially thought you were referring to one of
the HBase daemons, but if this is the MR job then whether or not it closes the
FileSystem is probably rather academic because it would close the FileSystem as
the JVM was about to shut down anyway, so it likely doesn't affect whether or
not you see a problem with memory usage.
* Do you have anything in hadoop-metrics2.properties at all?
> S3A Metrics in S3AInstrumentation Cause Memory Leaks
> ----------------------------------------------------
>
> Key: HADOOP-15392
> URL: https://issues.apache.org/jira/browse/HADOOP-15392
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.1.0
> Reporter: Voyta
> Priority: Major
>
> While using HBase S3A Export Snapshot utility we started to experience memory
> leaks of the process after version upgrade.
> By running code analysis we traced the cause to revision
> 6555af81a26b0b72ec3bee7034e01f5bd84b1564 that added the following static
> reference (singleton):
> private static MetricsSystem metricsSystem = null;
> When application uses S3AFileSystem instance that is not closed immediately
> metrics are accumulated in this instance and memory grows without any limit.
>
> Expectation:
> * It would be nice to have an option to disable metrics completely as this
> is not needed for Export Snapshot utility.
> * Usage of S3AFileSystem should not contain any static object that can grow
> indefinitely.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
