[
https://issues.apache.org/jira/browse/HDFS-13098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347783#comment-16347783
]
Íñigo Goiri commented on HDFS-13098:
------------------------------------
Currently, we do the assignment of DNs to subclusters using external tools that
generate {{hdfs-site.xml}}. These tools could be moved into the RBF
infrastructure.
I had some initial conversation about this topic with [~curino].
One of his concerns was to avoid passing every single heartbeat through the
Routers.
To solve this, we could make the DNs to just register the first time through
the Router and afterwards switch to heartbeating into the actual Namenodes.
I think this could also apply to YARN federation and we could share some
infrastructure; [~subru], [~giovanni.fumarola], any thoughts here?
Currently, this is an initial brainstorming and not much design done yet so
feedback is welcomed.
> RBF: Datanodes interacting with Routers
> ---------------------------------------
>
> Key: HDFS-13098
> URL: https://issues.apache.org/jira/browse/HDFS-13098
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Íñigo Goiri
> Priority: Major
>
> Datanodes talk to particular Namenodes. We could use the Router
> infrastructure for the Datanodes to register/heartbeating into them and the
> Routers would forward this to particular Namenodes. This would make the
> assignment of Datanodes to subclusters potentially more dynamic.
> The implementation would potentially include:
> * Router to implement part of DatanodeProtocol
> * Forwarding DN messages into Routers
> * Policies to assign datanodes to subclusters
> * Datanodes to make blockpool configuration dynamic
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
