[ 
https://issues.apache.org/jira/browse/HADOOP-6121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726160#action_12726160
 ] 

Scott Carey commented on HADOOP-6121:
-------------------------------------

If you really want to go very robust with the JobTracker, you could implement 
it by embedding ZooKeeper and using it for all the RPC coordination and state 
tracking.  This is a more natural fit than ZK for the NN, IMO.

A job could be a node in ZK, TaskTrackers would be able to see what work they 
should do by looking in specific nodes  -- and they can register their status 
and health as well.  Something like a Reduce job getting the map outputs it 
needs would simply be listing children from a reduce node for a job. 
Fault tolerance would come for free, and this form of RPC would likely be more 
efficient than the current one.

Furthermore, with watches firing off, there would be no ping-response sleep 
delays which would improve performance, and significantly improve latency for 
latency sensitive jobs.

> High Availability support for Hadoop
> ------------------------------------
>
>                 Key: HADOOP-6121
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6121
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: dfs, mapred
>            Reporter: Jie Qiu
>
> Currently, We look at the HA of Hadoop cluster. We need to consider the 
> NameNode HA as well as Jobtracker HA. For NameNode, we want to build 
> primary/standy or master-slaves pattern to provide NameNode HA. Therefore, we 
> need to consider how to ship log between primary/standby/slaves and how 
> commit "write" operation to NameNode after the agreement among 
> primary/standby/slaves on log. Whether will we use Linux HA package or 
> NameNode-built-in HA package without the help of outter Linux HA package. 
> After NameNode become high availability, is it necessary to provide HA for 
> Jobtracker? Can Jobtracker  persist the states of Jobs and tasks into HA 
> NameNode? Or Jobtracker also needs the same approach from NameNode for HA 
> support.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply via email to