[
https://issues.apache.org/jira/browse/YARN-5368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935302#comment-15935302
]
Varun Saxena commented on YARN-5368:
------------------------------------
[~jeagles], nice catch.
Closing of only the last iterator in the loop must be the reason for leak.
The leak we got in NM, albeit due to our private code, was also due to
DBIterator not being closed.
Using try-with-resources approach for DBIterator should be fine.
How about using try-with-resources for DBIterator elsewhere in the
RollingLevelDBTimelineStore class i.e. where it's not used in the loop, just to
make the code consistent.
> memory leak at timeline server
> ------------------------------
>
> Key: YARN-5368
> URL: https://issues.apache.org/jira/browse/YARN-5368
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
> Affects Versions: 2.7.1
> Environment: HDP2.4
> CentOS 6.7
> jdk1.8.0_72
> Reporter: Wataru Yukawa
> Assignee: Jonathan Eagles
> Attachments: YARN-5368.1.patch
>
>
> memory usage of timeline server machine increases gradually.
> https://gyazo.com/952dad96c77ae053bae2e4d8c8ab0572
> please check since April.
> According to my investigation, timeline server used about 25GB.
> top command result
> {code}
> 90577 yarn 20 0 28.4g 25g 12m S 0.0 40.1 5162:53
> /usr/java/jdk1.8.0_72/bin/java -Dproc_timelineserver -Xmx1024m
> -Dhdp.version=2.4.0.0-169 -Dhadoop.log.dir=/var/log/hadoop-yarn/yarn
> -Dyarn.log.dir=/var/log/hadoop-yarn/yarn ...
> {code}
> ps command result
> {code}
> $ ps ww 90577
> 90577 ? Sl 5162:53 /usr/java/jdk1.8.0_72/bin/java
> -Dproc_timelineserver -Xmx1024m -Dhdp.version=2.4.0.0-169
> -Dhadoop.log.dir=/var/log/hadoop-yarn/yarn
> -Dyarn.log.dir=/var/log/hadoop-yarn/yarn
> -Dhadoop.log.file=yarn-yarn-timelineserver-myhost.log
> -Dyarn.log.file=yarn-yarn-timelineserver-myhost.log -Dyarn.home.dir=
> -Dyarn.id.str=yarn -Dhadoop.root.logger=INFO,EWMA,RFA
> -Dyarn.root.logger=INFO,EWMA,RFA
> -Djava.library.path=:/usr/hdp/2.4.0.0-169/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.0.0-169/hadoop/lib/native:/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir:/usr/hdp/2.4.0.0-169/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.0.0-169/hadoop/lib/native:/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir
> -Dyarn.policy.file=hadoop-policy.xml
> -Djava.io.tmpdir=/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir
> -Dhadoop.log.dir=/var/log/hadoop-yarn/yarn
> -Dyarn.log.dir=/var/log/hadoop-yarn/yarn
> -Dhadoop.log.file=yarn-yarn-timelineserver-myhost.log
> -Dyarn.log.file=yarn-yarn-timelineserver-myhost.log
> -Dyarn.home.dir=/usr/hdp/current/hadoop-yarn-timelineserver
> -Dhadoop.home.dir=/usr/hdp/2.4.0.0-169/hadoop
> -Dhadoop.root.logger=INFO,EWMA,RFA -Dyarn.root.logger=INFO,EWMA,RFA
> -Djava.library.path=:/usr/hdp/2.4.0.0-169/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.0.0-169/hadoop/lib/native:/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir:/usr/hdp/2.4.0.0-169/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.0.0-169/hadoop/lib/native:/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir
> -classpath
> /usr/hdp/2.4.0.0-169/hadoop/conf:/usr/hdp/2.4.0.0-169/hadoop/conf:/usr/hdp/2.4.0.0-169/hadoop/conf:/usr/hdp/2.4.0.0-169/hadoop/lib/*:/usr/hdp/2.4.0.0-169/hadoop/.//*:/usr/hdp/2.4.0.0-169/hadoop-hdfs/./:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/*:/usr/hdp/2.4.0.0-169/hadoop-hdfs/.//*:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/*:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//*:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/*:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//*::/usr/hdp/2.4.0.0-169/tez/*:/usr/hdp/2.4.0.0-169/tez/lib/*:/usr/hdp/2.4.0.0-169/tez/conf:/usr/hdp/2.4.0.0-169/tez/*:/usr/hdp/2.4.0.0-169/tez/lib/*:/usr/hdp/2.4.0.0-169/tez/conf:/usr/hdp/current/hadoop-yarn-timelineserver/.//*:/usr/hdp/current/hadoop-yarn-timelineserver/lib/*:/usr/hdp/2.4.0.0-169/hadoop/conf/timelineserver-config/log4j.properties
>
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer
> {code}
>
> Alghough I set -Xmx1024m, actual memory usage is 25GB.
> After I restart timeline server, memory usage of timeline server machine
> decreases.
> https://gyazo.com/130600c17a7d41df8606727a859ae7e3
> Now timelineserver uses less than 1GB memory.
> top command result
> {code}
> 6163 yarn 20 0 3959m 783m 46m S 0.3 1.2 3:37.60
> /usr/java/jdk1.8.0_72/bin/java -Dproc_timelineserver -Xmx1024m
> -Dhdp.version=2.4.0.0-169 ...
> {code}
> I suspect memory leak at timeline server.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
