[
https://issues.apache.org/jira/browse/HDDS-11505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krishna Kumar Asawa reassigned HDDS-11505:
------------------------------------------
Assignee: Sadanand Shenoy
> Remove usage of OzoneConfiguration in XceiverClientMetrics.
> -----------------------------------------------------------
>
> Key: HDDS-11505
> URL: https://issues.apache.org/jira/browse/HDDS-11505
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Sadanand Shenoy
> Assignee: Sadanand Shenoy
> Priority: Major
>
> Encountered a case where not closing the Ozone client lead to memory leak
> related to the usage of OzoneConfiguration objects.
> This was added via HDDS-10182
> OzoneConfiguration is a bulky object which contains a hashmap having a lot of
> properties.
> In XceiverClientMetrics.java we instantiate the conf object only to fetch one
> property. If possible this should be removed and passed as a member variable
> instead.
> {code:java}
> private OzoneConfiguration conf = new OzoneConfiguration();
> private int[] intervals = conf.getInts(OzoneConfigKeys
> .OZONE_XCEIVER_CLIENT_METRICS_PERCENTILES_INTERVALS_SECONDS_KEY); {code}
> From the memory leak report in an app creating multiple unclosed client
> instances .
> {quote}"1,663 instances of
> {*}org.apache.hadoop.hdds.conf.OzoneConfiguration{*}, loaded by
> *com.xxxx.UrlClassLoader @ 0xd05fd5c0* occupy *650,830,032 (61.63%)* bytes.
> Most of these instances are referenced from one instance of
> {*}java.util.HashMap$Node[]{*}, loaded by {*}<system class loader>{*}, which
> occupies *69,680 (0.01%)* bytes. The instance is referenced by
> {*}org.apache.hadoop.metrics2.impl.MetricsSystemImpl @ 0xd1453c38{*}, loaded
> by *com.xxxx.UrlClassLoader* {*}@ 0xd05fd5c0{*}.
> Thread *java.util.TimerThread @ 0xd12ff010 Timer for 'XceiverClientMetrics'
> metrics system* has a local variable or reference to
> *org.apache.hadoop.metrics2.impl.MetricsSystemImpl$4 @ 0xd12ff1a8* which is
> on the shortest path to {*}java.util.HashMap$Node[4096] @ 0xfacaa5f8{*}. The
> thread *java.util.TimerThread @ 0xd12ff010 Timer for 'XceiverClientMetrics'
> metrics system* keeps local variables with total size *464 (0.00%)* bytes.
> Significant stack frames and local variables
> * java.util.TimerThread.mainLoop()V (Timer.java:553)
> ** org.apache.hadoop.metrics2.impl.MetricsSystemImpl$4 @ 0xd12ff1a8 retains
> 56 (0.00%) bytes{quote}
> How does the leak happen?
> 1. The app keeps on creating new Ozone clients but does not close it
> 2. Each client object creates an XceiverClientMetrics object that holds a
> reference to OzoneConfiguration object and registers this metric with the
> MetricsRegister.
> 3. Even after the client is done with its task, the conf object is still
> referenced as the XceiverClientMetrics object is not unregistered and
> metricsSystem still tracks the metric and holds an active reference to the
> conf object.
> Closing the client will unregister the metric removing the active reference
> but we should eliminate usage if possible
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
