Swapnil Daingade created MYRIAD-127:
---------------------------------------
Summary: Recommend nodes to launch NodeManagers optimized for
locality based on input data for Yarn jobs
Key: MYRIAD-127
URL: https://issues.apache.org/jira/browse/MYRIAD-127
Project: Myriad
Issue Type: New Feature
Reporter: Swapnil Daingade
Assignee: Swapnil Daingade
Hadoop/Yarn works on the principle of taking computation to the data. Thus data
locality is important for getting optimal performance. When a yarn job is
launched, the user specifies the dfs location of the data being operated on.
Looking at all these locations (from various running yarn jobs) being operated
on, we can try to predict the best location for launching NodeManagers
optimized for locality.
For. e.g We have a 20 node mesos cluster and a user has a 5 node yarn cluster
running a few jobs. Looking at the data being operated on by the yarn jobs, we
can come up with a recommendation for which 5 nodes to launch NodeManagers on
optimized for locality.
In a more autonomous mode, we could flex down some of the existing NM's and
flexup new ones on new nodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
