Varun Vasudev updated YARN-160:

    Attachment: apache-yarn-160.0.patch

Patch to automatically calculate cpu and memory from the OS. In case of cpu, 
I've added a flag to allow admins to decide if they want to count hyperthreads 
as vcores or not(by default they are counted as vcores). The flag currently 
works on Linux only.

In case of memory, I calculate the memory for containers as 80% of (RAM - 
2*HADOOP_HEAPSIZE), to account for memory used by the datanode and the 

I've also changed the default behaviour to use the calculated values instead of 
8 and 8192 for cpu and memory that we have been using till now. 

Feedback would be welcome.

> nodemanagers should obtain cpu/memory values from underlying OS
> ---------------------------------------------------------------
>                 Key: YARN-160
>                 URL: https://issues.apache.org/jira/browse/YARN-160
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager
>    Affects Versions: 2.0.3-alpha
>            Reporter: Alejandro Abdelnur
>            Assignee: Varun Vasudev
>             Fix For: 2.6.0
>         Attachments: apache-yarn-160.0.patch
> As mentioned in YARN-2
> *NM memory and CPU configs*
> Currently these values are coming from the config of the NM, we should be 
> able to obtain those values from the OS (ie, in the case of Linux from 
> /proc/meminfo & /proc/cpuinfo). As this is highly OS dependent we should have 
> an interface that obtains this information. In addition implementations of 
> this interface should be able to specify a mem/cpu offset (amount of mem/cpu 
> not to be avail as YARN resource), this would allow to reserve mem/cpu for 
> the OS and other services outside of YARN containers.

This message was sent by Atlassian JIRA

Reply via email to