[
https://issues.apache.org/jira/browse/OOZIE-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16602847#comment-16602847
]
Peter Bacsko commented on OOZIE-3307:
-------------------------------------
That's right, we do set {{-Xmx}} automatically in MR-based Oozie:
[https://github.com/apache/oozie/blob/branch-4.3/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L382-L415]
It's missing from the Oozie-on-YARN based implementation. Perhaps, based on the
resource request, we could limit {{-Xmx}} as 80-90 % of the requested memory
to make sure that OOME happens before the container is killyed by YARN.
> oozie workflow gets failed throwing error virtual memory limits
> ---------------------------------------------------------------
>
> Key: OOZIE-3307
> URL: https://issues.apache.org/jira/browse/OOZIE-3307
> Project: Oozie
> Issue Type: Bug
> Affects Versions: 5.0.0
> Reporter: Sabir Naikwadi
> Priority: Critical
>
> Application application_1531909575787_0039 failed 2 times due to AM Container
> for appattempt_1531909575787_0039_000002 exited with exitCode: -103
> Failing this attempt.Diagnostics: Container
> [pid=11516,containerID=container_1531909575787_0039_02_000001] is running
> beyond virtual memory limits. Current usage: 469.8 MB of 2 GB physical memory
> used; 10.0 GB of 10 GB virtual memory used. Killing container.
> Dump of the process-tree for container_1531909575787_0039_02_000001 :
> | - PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE|
> | - 11516 11514 11516 11516 (bash) 1 3 115863552 682 /bin/bash -c
> /usr/lib/jvm/java-openjdk/bin/java
> -Dlog4j.configuration=container-log4j.properties -Dlog4j.debug=true
> -Dyarn.app.container.log.dir=/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_000001
> -Dyarn.app.container.log.filesize=1048576 -Dhadoop.root.logger=INFO,CLA
> -Dhadoop.root.logfile=syslog -Dsubmitter.user=dev
> org.apache.oozie.action.hadoop.LauncherAM
> 1>/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_000001/stdout
>
> 2>/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_000001/stderr|
> | - 11755 11516 11516 11516 (java) 1142 71 10658242560 119576
> /usr/lib/jvm/java-openjdk/bin/java
> -Dlog4j.configuration=container-log4j.properties -Dlog4j.debug=true
> -Dyarn.app.container.log.dir=/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_000001
> -Dyarn.app.container.log.filesize=1048576 -Dhadoop.root.logger=INFO,CLA
> -Dhadoop.root.logfile=syslog -Dsubmitter.user=dev
> org.apache.oozie.action.hadoop.LauncherAM
> Container killed on request. Exit code is 143
> Container exited with a non-zero exit code 143
> For more detailed output, check the application tracking page:
> [http://ip-10-20-201-36.us-gov-west-1.compute.internal:8088/cluster/app/application_1531909575787_0039]
> Then click on links to logs of each attempt.
> . Failing the application.|
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
