[ 
https://issues.apache.org/jira/browse/HDFS-10467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511831#comment-15511831
 ] 

Subru Krishnan commented on HDFS-10467:
---------------------------------------

Thanks [~jakace] and [~goiri] for the refactored patch. I made a quick pass in 
the context of HADOOP-13378 and I think that we can represent the YARN 
{{FederationStateStore}} using the generic {{StateStoreDriver}} you guys have 
defined. Personally I prefer the push mechanism we have in YARN-3671 as it's 
much simpler than the pull mechanism proposed here though I do agree both 
achieve the same result.

A couple of comments based on my quick scan:
  * We should add versioning to the generic {{StateStoreDriver}}. Refer 
[FederationStateStore|https://github.com/apache/hadoop/blob/YARN-2915/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/FederationStateStore.java].
  * Use _jcache_ instead of writing custom key-caches based on 
_ConcurrentHashMaps_. In fact, I feel we can refactor the (jcache-based) cache 
in 
[FederationStateStoreFacade|https://github.com/apache/hadoop/blob/YARN-2915/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/utils/FederationStateStoreFacade.java]
 and use it across both efforts.
  * Lastly I would suggest using [Curator|http://curator.apache.org/] for 
*ZooKeeper* implementation as we have moved to it in YARN (YARN-4438 and follow 
up work).
 

> Router-based HDFS federation
> ----------------------------
>
>                 Key: HDFS-10467
>                 URL: https://issues.apache.org/jira/browse/HDFS-10467
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 2.7.2
>            Reporter: Inigo Goiri
>            Assignee: Inigo Goiri
>         Attachments: HDFS Router Federation.pdf, HDFS-10467.PoC.001.patch, 
> HDFS-10467.PoC.patch, HDFS-Router-Federation-Prototype.patch
>
>
> Add a Router to provide a federated view of multiple HDFS clusters.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

Reply via email to