[
https://issues.apache.org/jira/browse/YARN-5523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421459#comment-15421459
]
Xuan Gong commented on YARN-5523:
---------------------------------
We should use stream output instead of directly calling println which can avoid
this oom issue.
The patch did not add any unit test cases given that we did not have any
functional unit tests for fetching logs from running container. But I did the
manual test to make sure the fix works.
1) Run sleep job
2) override YARN_CLIENT_OPTS and set a very low Xmx value
3) run yarn logs command (without the fix), and we could see the exact oom
exception as shown in the description
4) run yarn logs command (with the fix), and we could not see the oom exception
anymore.
> Yarn live log aggregation runs out of memory
> --------------------------------------------
>
> Key: YARN-5523
> URL: https://issues.apache.org/jira/browse/YARN-5523
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
> Affects Versions: 2.9.0
> Reporter: Prasanth Jayachandran
> Assignee: Xuan Gong
> Attachments: YARN-5523.1.patch
>
>
> Fetching a 256MB log from container caused OOM on client
> {code}
> [hive@ctr-e20-1468887904486-0007-01-000003 ~]$ yarn logs -applicationId
> application_1470931023753_0001 -log_files log_2016-08-11-21_3.done > logs
> 16/08/11 21:58:11 INFO impl.TimelineClientImpl: Timeline service address:
> http://ctr-e20-1468887904486-0007-01-000003.hwx.site:8188/ws/v1/timeline/
> 16/08/11 21:58:11 INFO client.RMProxy: Connecting to ResourceManager at
> ctr-e20-1468887904486-0007-01-000003.hwx.site/172.27.8.192:8050
> 16/08/11 21:58:12 INFO client.AHSProxy: Connecting to Application History
> server at ctr-e20-1468887904486-0007-01-000003.hwx.site/172.27.8.192:10200
> Can not find any log file matching the pattern: [log_2016-08-11-21_3.done]
> for the container: container_e04_1470931023753_0001_01_000001 within the
> application: application_1470931023753_0001
> Can not find any log file matching the pattern: [log_2016-08-11-21_3.done]
> for the container: container_e04_1470931023753_0001_01_000002 within the
> application: application_1470931023753_0001
> Can not find any log file matching the pattern: [log_2016-08-11-21_3.done]
> for the container: container_e04_1470931023753_0001_01_000003 within the
> application: application_1470931023753_0001
> Can not find any log file matching the pattern: [log_2016-08-11-21_3.done]
> for the container: container_e04_1470931023753_0001_01_000004 within the
> application: application_1470931023753_0001
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:3332)
> at
> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
> at
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:569)
> at java.lang.StringBuilder.append(StringBuilder.java:190)
> at
> com.sun.jersey.core.util.ReaderWriter.readFromAsString(ReaderWriter.java:172)
> at
> com.sun.jersey.core.util.ReaderWriter.readFromAsString(ReaderWriter.java:157)
> at
> com.sun.jersey.core.provider.AbstractMessageReaderWriterProvider.readFromAsString(AbstractMessageReaderWriterProvider.java:114)
> at
> com.sun.jersey.core.impl.provider.entity.StringProvider.readFrom(StringProvider.java:73)
> at
> com.sun.jersey.core.impl.provider.entity.StringProvider.readFrom(StringProvider.java:58)
> at
> com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:553)
> at
> com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:506)
> at
> org.apache.hadoop.yarn.client.cli.LogsCLI.printContainerLogsFromRunningApplication(LogsCLI.java:477)
> at
> org.apache.hadoop.yarn.client.cli.LogsCLI.fetchApplicationLogs(LogsCLI.java:950)
> at org.apache.hadoop.yarn.client.cli.LogsCLI.runCommand(LogsCLI.java:280)
> at org.apache.hadoop.yarn.client.cli.LogsCLI.run(LogsCLI.java:102)
> at org.apache.hadoop.yarn.client.cli.LogsCLI.main(LogsCLI.java:307)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
