Inline. On Sat, Aug 29, 2015 at 9:45 AM, Muthu Ganesh <[email protected]> wrote:
> Hi, > > Why does YARN block an entire resource once a container has been allocated > in the node? A job might use different quantities of resources at different > points of time and allocating a pre-determined set of resources might be > counter-productive. Yes and no, depends on what is more important to the user - utilization or predictable performance. Admittedly, Yarn could consider node utilization while scheduling containers to increase utilization without sacrificing predictable performance. YARN-1011 and a couple of other JIRAs target this, and could use contributions. :) > For example, if the resources requested by the job are > 1VCORE and 4GB of RAM, the entire 4GB is blocked (other Jobs/processes can > not access this memory) irrespective of whether or not the job uses the > complete 4GB at a given point of time. > > This might be a problem with I/O bound tasks wherein the CPU utilization is > close to 1% but 1VCORE is used nevertheless since the hdfs-site.xml > dictates a minimum VCORE of 1. This VCORE could have been given to a CPU > intensive job wherein the 1VCORE allocation actually makes sense. > If the goal is to get fractional cpus, one could always bump up the number of virtual cores on a node so each virtual core represents a smaller fraction of a physical core and use that to schedule. > > The YARN-1197 <https://issues.apache.org/jira/browse/YARN-1197> Jira shows > support for dynamic containers but I believe the container sizes are not > increased/decreased fractionally. > > Thanks, > Muthu >
