[
https://issues.apache.org/jira/browse/YARN-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14569237#comment-14569237
]
Jason Lowe commented on YARN-3758:
----------------------------------
First off, one should never set the heap size and the container size to the
same value. The container size needs to be big enough to hold the entire
process, not just the heap, so it needs to also consider the overhead of the
JVM itself and any off-heap usage (e.g.: JVM code, data, thread stacks, shared
libs, off-heap allocations, etc.). If you set the heap size to the same size
as the container then when the heap fills up the process overall will be bigger
than the heap size and YARN will kill the container.
Couple of things to check:
- Does the job configuration show that it is indeed asking for only 256 MB
containers for tasks? Check the job configuration link for the job on the job
history server or the configuration link for the AM's UI while the job is
running.
- Check the RM logs to verify what minimum allocation size it is loading from
the configs and what request size it is allocating per task
> The mininum memory setting(yarn.scheduler.minimum-allocation-mb) is not
> working in container
> --------------------------------------------------------------------------------------------
>
> Key: YARN-3758
> URL: https://issues.apache.org/jira/browse/YARN-3758
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 2.4.0
> Reporter: skrho
>
> Hello there~~
> I have 2 clusters
> First cluster is 5 node , default 1 application queue, Capacity scheduler, 8G
> Physical memory each node
> Second cluster is 10 node, 2 application queuey, fair-scheduler, 230G
> Physical memory each node
> Wherever a mapreduce job is running, I want resourcemanager is to set the
> minimum memory 256m to container
> So I was changing configuration in yarn-site.xml & mapred-site.xml
> yarn.scheduler.minimum-allocation-mb : 256
> mapreduce.map.java.opts : -Xms256m
> mapreduce.reduce.java.opts : -Xms256m
> mapreduce.map.memory.mb : 256
> mapreduce.reduce.memory.mb : 256
> In First cluster whenever a mapreduce job is running , I can see used memory
> 256m in web console( http://installedIP:8088/cluster/nodes )
> But In Second cluster whenever a mapreduce job is running , I can see used
> memory 1024m in web console( http://installedIP:8088/cluster/nodes )
> I know default memory value is 1024m, so if there is not changing memory
> setting, the default value is working.
> I have been testing for two weeks, but I don't know why mimimum memory
> setting is not working in second cluster
> Why this difference is happened?
> Am I wrong setting configuration?
> or Is there bug?
> Thank you for reading~~
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
