[
https://issues.apache.org/jira/browse/YARN-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300766#comment-16300766
]
Vinod Kumar Vavilapalli commented on YARN-7284:
-----------------------------------------------
Turning off DEBUG for ContainerLocalizer should avoid this problem.
But presumably it was turned on to debug some issues with localization.
It's high time we send the ContainerLocalizer output also to a file that
eventually gets aggregated. The key challenge is that ContainerLocalizer isn't
tied to one container - it can download resources for multiple containers. So,
tying to log-aggregation will need some non-trivial tricks.
> NodeManager crashes with OOM when Debug log enabled for ContainerLocalizer
> ---------------------------------------------------------------------------
>
> Key: YARN-7284
> URL: https://issues.apache.org/jira/browse/YARN-7284
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 2.7.3
> Reporter: Prabhu Joseph
> Attachments: Screen Shot 2017-10-03 at 1.29.35 PM.png, Screen Shot
> 2017-10-03 at 1.29.48 PM.png
>
>
> NodeManager crashes with OOM when DEBUG log enabled for ContainerLocalizer.
> {code}
> 2017-10-03 07:25:20,066 FATAL yarn.YarnUncaughtExceptionHandler
> (YarnUncaughtExceptionHandler.java:uncaughtException(51)) - Thread
> Thread[Thread-2114,5,main] threw an Error. Shutting down now...
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:3332)
> at
> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
> at
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
> at
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421)
> at java.lang.StringBuffer.append(StringBuffer.java:272)
> at org.apache.hadoop.util.Shell$1.run(Shell.java:900)
> {code}
> errThread part of Hadoop Common Shell reads all the DEBUG log lines and
> appends to StringBuffer errMsg. As per the heap dump, the errMsg stores more
> than 1GB of contents. (attached image)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
