[
https://issues.apache.org/jira/browse/HDFS-10467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15304668#comment-15304668
]
Inigo Goiri commented on HDFS-10467:
------------------------------------
This approach should support the scenarios in the mail thread. Right now we
support the most typical operations of the RPC interface and the basic ones for
REST. We don't do proxying of the requests to the DNs as HttpFs does but at
some point we might; for now we just point HttpFs to our Routers. Instead of
extending HttpFs we went into mimicing the NN to provide the full interface
(i.e., RPC) and mimic the image of a big large NN.
The main difference with the approach proposed in the mail thread is the
addition of the State Store as a centralized storage for the federation state.
This mimics the architecture from YARN federation in YARN-2915.
If the document is not enough, I can provide a patch with the full approach
(including Router, State Store and a simple cluster rebalancer). I think later
on, we should split it into multiple subtasks.
> Router-based HDFS federation
> ----------------------------
>
> Key: HDFS-10467
> URL: https://issues.apache.org/jira/browse/HDFS-10467
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: fs
> Affects Versions: 2.7.2
> Reporter: Inigo Goiri
> Attachments: HDFS Router Federation.pdf
>
>
> Add a Router to provide a federated view of multiple HDFS clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
