[
https://issues.apache.org/jira/browse/HADOOP-13378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15378732#comment-15378732
]
Vinayakumar B edited comment on HADOOP-13378 at 7/15/16 11:50 AM:
------------------------------------------------------------------
It looks like the main two candidates to be shared between YARN and HDFS
Router-based federation are:
# Generic State Store: Right now, in HDFS-10647 we make use of a generic State
Store that stores entities. Once we want to store a new piece of information,
we just need to create a new serialization/deserialization method. I'm pretty
sure we could abstract this even further and it would be fairly easy to use in
YARN. Actually, I think this could be even extended to components like the
{{RMStateStore}}.
# Membership maintenance: Our proposal in HDFS-10467 is to follow a pull model
to avoid changes in the Namenode. However, this brings some additional problems
like fault tolerance: the Router can die while the Namenode is still running.
To mitigate this, we made Routers able to monitor multiple Namenodes. Then,
when the Routers get the information from the State Store, they do use quorum
to unify views. Not sure if it makes sense for YARN federation to use this
approach but it's worth considering. The other challenge would be to make this
service abstract enough.
[~subru], [~curino], [~giovanni.fumarola], any thoughts on this?
was (Author: elgoiri):
It looks like the main two candidates to be shared between YARN and HDFS
Router-based federation are:
# Generic State Store: Right now, in HDFS-10647 we make use of a generic State
Store that stores entities. Once we want to store a new piece of information,
we just need to create a new serialization/deserialization method. I'm pretty
sure we could abstract this even further and it would be fairly easy to use in
YARN. Actually, I think this could be even extended to components like the
{{RMStateStore}}.
# Membership maintenance: Our proposal in HDFS-10647 is to follow a pull model
to avoid changes in the Namenode. However, this brings some additional problems
like fault tolerance: the Router can die while the Namenode is still running.
To mitigate this, we made Routers able to monitor multiple Namenodes. Then,
when the Routers get the information from the State Store, they do use quorum
to unify views. Not sure if it makes sense for YARN federation to use this
approach but it's worth considering. The other challenge would be to make this
service abstract enough.
[~subru], [~curino], [~giovanni.fumarola], any thoughts on this?
> Common features between YARN and HDFS Router-based federation
> -------------------------------------------------------------
>
> Key: HADOOP-13378
> URL: https://issues.apache.org/jira/browse/HADOOP-13378
> Project: Hadoop Common
> Issue Type: New Feature
> Reporter: Inigo Goiri
>
> HDFS-10647 uses a similar architecture to the one proposed in YARN-2915. This
> JIRA tries to identify what is common between these two efforts and try to
> build a common framework.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
