[
https://issues.apache.org/jira/browse/YARN-168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13633110#comment-13633110
]
Hitesh Shah commented on YARN-168:
----------------------------------
As started in YARN-470, this is fixed for 2.0.4-alpha. Refer to
http://hadoop.markmail.org/thread/mfaywrkctaezvlrv.
> No way to turn off virtual memory limits without turning off physical memory
> limits
> -----------------------------------------------------------------------------------
>
> Key: YARN-168
> URL: https://issues.apache.org/jira/browse/YARN-168
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 3.0.0
> Reporter: Harsh J
>
> Asked and reported by a user (Krishna) on ML:
> {quote}
> This is possible to do, but you've hit a bug with the current YARN
> implementation. Ideally you should be able to configure the vmem-pmem
> ratio (or an equivalent config) to be -1, to indicate disabling of
> virtual memory checks completely (and there's indeed checks for this),
> but it seems like we are enforcing the ratio to be at least 1.0 (and
> hence negatives are disallowed).
> You can't workaround by setting the NM's offered resource.mb to -1
> either, as you'll lose out on controlling maximum allocations.
> Please file a YARN bug on JIRA. The code at fault lies under
> ContainersMonitorImpl#init(…).
> On Thu, Oct 18, 2012 at 4:00 PM, Krishna Kishore Bonagiri
> <[email protected]> wrote:
> > Hi,
> >
> > Is there a way we can ask the YARN RM for not killing a container when it
> > uses excess virtual memory than the maximum it can use as per the
> > specification in the configuration file yarn-site.xml? We can't always
> > estimate the amount of virtual memory needed for our application running on
> > a container, but we don't want to get it killed in a case it exceeds the
> > maximum limit.
> >
> > Please suggest as to how can we come across this issue.
> >
> > Thanks,
> > Kishore
> {quote}
> Basically, we're doing:
> {code}
> // ///////// Virtual memory configuration //////
> float vmemRatio = conf.getFloat(
> YarnConfiguration.NM_VMEM_PMEM_RATIO,
> YarnConfiguration.DEFAULT_NM_VMEM_PMEM_RATIO);
> Preconditions.checkArgument(vmemRatio > 0.99f,
> YarnConfiguration.NM_VMEM_PMEM_RATIO +
> " should be at least 1.0");
> this.maxVmemAllottedForContainers =
> (long)(vmemRatio * maxPmemAllottedForContainers);
> {code}
> For virtual memory monitoring to be disabled, maxVmemAllottedForContainers
> has to be -1. For that to be -1, given the above buggy computation, vmemRatio
> must be -1 or maxPmemAllottedForContainers must be -1.
> If vmemRatio were -1, we fail the precondition check and exit.
> If maxPmemAllottedForContainers, we also end up disabling physical memory
> monitoring.
> Or perhaps that makes sense - to disable both physical and virtual memory
> monitoring, but that way your NM becomes infinite in resource grants, I think.
> We need a way to selectively disable kills done via virtual memory
> monitoring, which is the base request here.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
