[
https://issues.apache.org/jira/browse/YARN-10047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17002521#comment-17002521
]
Wilfred Spiegelenburg commented on YARN-10047:
--
Accounting is not done by YARN, it is done by the OS. It sounds like you are
saying that the OS is not accounting correctly.
>From a YARN perspective we consider any process that is forked from the
>container script to be part of the container. All sub processes are therefore
>part of the container. You must account for that inside the container.
If you are referring to some other issue or I am not understanding the problem
you see correctly then please show it with some logs or screenshots.
> Memory consume of process tree will consider subprocess which may make
> container exit unexcepted
> -
>
> Key: YARN-10047
> URL: https://issues.apache.org/jira/browse/YARN-10047
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Reporter: zhoukang
>Assignee: zhoukang
>Priority: Major
>
> As below, we have a case which spark driver execute some scripts.Then
> sometimes the driver will be killed.
> {code:java}
> yarn.174410.log.2019-12-17.02:2019-12-17,06:59:14,831 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Container
> [pid=50529,containerID=container_e917_1576303656075_174957_01_003197] is
> running beyond physical memory limits. Current usage: 50.28 GB of 5.25 GB
> physical memory used; xxx. Killing container.
> {code}
> {code:java}
> boolean isProcessTreeOverLimit(String containerId,
> long currentMemUsage,
> long curMemUsageOfAgedProcesses,
> long vmemLimit) {
> boolean isOverLimit = false;
>
> /**
> if (currentMemUsage > (2 * vmemLimit)) {
> LOG.warn("Process tree for container: " + containerId
> + " running over twice " + "the configured limit. Limit=" +
> vmemLimit
> + ", current usage = " + currentMemUsage);
> isOverLimit = true;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
