[
https://issues.apache.org/jira/browse/MAPREDUCE-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12997589#comment-12997589
]
Scott Carey commented on MAPREDUCE-279:
---------------------------------------
Good stuff!
Does the NodeManager communicate to the ResourceManager similar to now (ping ->
response RPC)? I ask because some of the bottlenecks and complexities now are
due to this style of RPC. I've changed a couple systems in the past from ping
-> response to register -> callback in the past and these became more efficient
and the code became simpler. With ZooKeeper in there, I wonder how much of the
communicaton now uses ZooKeeper watches for efficiency and low latency.
When a Job starts up in the ApplicationMaster, does it have to wait for pings
to get resources from the scheduler? Or is the data all there in ZK, so that
ramp-up times for jobs is much faster and resource reassignment for jobs with
short lived tasks isn't completely throttled by the rate of pings?
In any case, the new architecture is decoupled and it should be much easier to
make enhancements with this separation.
> Map-Reduce 2.0
> --------------
>
> Key: MAPREDUCE-279
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-279
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker, tasktracker
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Fix For: 0.23.0
>
>
> Re-factor MapReduce into a generic resource scheduler and a per-job,
> user-defined component that manages the application execution.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
