Thanks for the reply, Varun. So if I use the DefaultContainerExecutor and run a C++ application via a shell script inside a container whose virtual memory limit is, for example, 2 GB, and that application does a malloc for 3 GB, YARN will kill the container? I always just thought that YARN kept its eye on the JVM it spins up for the container (under the DefaultContainerExecutor).
-Kevin On Mon, May 25, 2015 at 4:17 AM, Varun Vasudev <[email protected]> wrote: > Hi Kevin, > > By default, the NodeManager monitors physical and virtual memory usage > of containers. Containers that exceed either limit are killed. Admins can > disable the checks by setting yarn.nodemanager.pmem-check-enabled and/or > yarn.nodemanager.vmem-check-enabled > to false. The virtual memory limit for a container is determined using the > config variable yarn.nodemanager.vmem-pmem-ratio(default value is 2.1). > > In case of vcores - > > 1. If you’re using Cgroups under LinuxContainerExecutor, by default, > if there is spare CPU available on the node, your container will be allowed > to use it. Admins can restrict containers to use only the CPU allocated to > them by setting > yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage > to true. This setting is only applicable when using Cgroups under > LinuxContainerExecutor. > 2. If you aren’t using Cgroups under LinuxContainerExecutor, there is > no limiting of the amount of the CPU that containers can use. > > -Varun > > From: Kevin > Reply-To: "[email protected]" > Date: Friday, May 22, 2015 at 3:30 AM > To: "[email protected]" > Subject: Using YARN with native applications > > Hello, > > I have been using the distributed shell application and Oozie to run > native C++ applications in the cluster. Is YARN able to see the resources > these native applications use. For example, if I use Oozie's shell action, > the NodeManager hosts the mapper container and allocates a certain amount > of memory and vcores (as configured). What happens if my C++ application > uses more memory or vcores than the NodeManager allocated? > > I was looking in the Hadoop code and I couldn't find my way to answer. > Although, it seems the LinuxContainerExecutor may be the answer to my > question since it uses cgroups. > > I'm interested to know how YARN reacts to non-Java applications running > inside of it. > > Thanks, > Kevin >
