[
https://issues.apache.org/jira/browse/HDFS-10467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15381726#comment-15381726
]
Ming Ma commented on HDFS-10467:
--------------------------------
[~elgoiri], nice work! Here are couple more questions about the design. I will
post code review questions in sub jiras.
* Support for mergeFs HADOOP-8298. We should be able to extend the design to
support this.There might be some issues around how to provision a new sub
folder (which namespace should own that) and how it works with rebalancer. This
could be a good addition for future work section.
* Handling of inconsistent state. Given routers cache which namenodes are
active, the state could be different from the actual namenode at that moment.
Thus routers might get {{StandbyException}} and need to retry on another
namenode. If so, does it mean the routers should leverage ipc
{{FailoverOnNetworkExceptionRetry}} or use DFSClient with hint for active
namenode?
* Soft state vs hard state. while subcluster active namenode machine and
load/space are soft state that can be reconstructed from namenodes; mount table
is hard state that need to be persisted. Is there any benefit separating them
out to use different state stores as they have different persistence
requirement, access patterns(mount table does't change much while load/space
update is frequent) and admin interface? For example, admin might want to
update mount table on demand; but not load/space state.
* Usage of subcluster load/space state. Is it correct that the only consumer of
subcluster's load/space state is the rebalancer? I image initially we would run
rebalancer manually. For that, the rebalancer can just pull subcluster's
load/space state from namenodes on demand. Then we don't have to store
subcluster load/space state in state store.
* Admin's modification of mount table. Besides rebalancer, admin might want to
update mount table during cluster initial setup as well as addition of new
namespace with new mount entry. If we continue to use mounttable.xml, then
admins can push the update the same way as viewFs setup. If we use ZK store,
them we need to provide tools to update state store.
* What is the performance optimization in your latest patch, based on async RPC
client?
> Router-based HDFS federation
> ----------------------------
>
> Key: HDFS-10467
> URL: https://issues.apache.org/jira/browse/HDFS-10467
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: fs
> Affects Versions: 2.7.2
> Reporter: Inigo Goiri
> Assignee: Inigo Goiri
> Attachments: HDFS Router Federation.pdf, HDFS-10467.PoC.001.patch,
> HDFS-10467.PoC.patch, HDFS-Router-Federation-Prototype.patch
>
>
> Add a Router to provide a federated view of multiple HDFS clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
