[
https://issues.apache.org/jira/browse/HDFS-10467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511831#comment-15511831
]
Subru Krishnan edited comment on HDFS-10467 at 9/22/16 2:00 AM:
----------------------------------------------------------------
Thanks [~jakace] and [~goiri] for the refactored patch. I made a quick pass in
the context of HADOOP-13378 and I think that we can represent the YARN
{{FederationStateStore}} using the generic {{StateStoreDriver}} you guys have
defined. Personally I prefer the push mechanism we have in YARN-3671 as it's
much simpler than the pull mechanism proposed here though I do agree both
achieve the same result.
A couple of comments based on my quick scan:
* We should add versioning to the generic {{StateStoreDriver}}. Refer
[FederationStateStore|https://github.com/apache/hadoop/blob/YARN-2915/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/FederationStateStore.java].
* Use _jcache_ instead of writing custom key-caches based on
_ConcurrentHashMaps_. In fact, I feel we can refactor the (jcache-based) cache
in
[FederationStateStoreFacade|https://github.com/apache/hadoop/blob/YARN-2915/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/utils/FederationStateStoreFacade.java]
and use it across both efforts.
* We should reuse the *RecordFactory* and supporting infrastructure from YARN
as opposed to coming up with a parallel structure in HDFS.
* Lastly I would suggest using [Curator|http://curator.apache.org/] for
*ZooKeeper* implementation as we have moved to it in YARN (YARN-4438 and follow
up work).
was (Author: subru):
Thanks [~jakace] and [~goiri] for the refactored patch. I made a quick pass in
the context of HADOOP-13378 and I think that we can represent the YARN
{{FederationStateStore}} using the generic {{StateStoreDriver}} you guys have
defined. Personally I prefer the push mechanism we have in YARN-3671 as it's
much simpler than the pull mechanism proposed here though I do agree both
achieve the same result.
A couple of comments based on my quick scan:
* We should add versioning to the generic {{StateStoreDriver}}. Refer
[FederationStateStore|https://github.com/apache/hadoop/blob/YARN-2915/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/FederationStateStore.java].
* Use _jcache_ instead of writing custom key-caches based on
_ConcurrentHashMaps_. In fact, I feel we can refactor the (jcache-based) cache
in
[FederationStateStoreFacade|https://github.com/apache/hadoop/blob/YARN-2915/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/utils/FederationStateStoreFacade.java]
and use it across both efforts.
* Lastly I would suggest using [Curator|http://curator.apache.org/] for
*ZooKeeper* implementation as we have moved to it in YARN (YARN-4438 and follow
up work).
> Router-based HDFS federation
> ----------------------------
>
> Key: HDFS-10467
> URL: https://issues.apache.org/jira/browse/HDFS-10467
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: fs
> Affects Versions: 2.7.2
> Reporter: Inigo Goiri
> Assignee: Inigo Goiri
> Attachments: HDFS Router Federation.pdf, HDFS-10467.PoC.001.patch,
> HDFS-10467.PoC.patch, HDFS-Router-Federation-Prototype.patch
>
>
> Add a Router to provide a federated view of multiple HDFS clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
