[
https://issues.apache.org/jira/browse/HDFS-13098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16350767#comment-16350767
]
Íñigo Goiri commented on HDFS-13098:
------------------------------------
bq. Do these subclusters also dynamically generated? That is say, we have N
datanodes, then these datanodes communicate with the Router. Based on the
assignment policies, the Router decides which subcluster the datanode belong to.
We could add new subclusters (i.e., new namenodes) and then we can use
consistent hashing to just move a small subset of the servers to the new
subcluster.
bq. But if the machines are moving across subclusters, how we plan to do?
Decommission in original subclsuter then add into new subcluster?
For now we would do this change of subclusters for servers that are empty: new
ones or servers that get reimaged/decommissioned.
Once we support rebalancing, we could merge both things.
At the beginning, we can leverage regular HDFS federation and the DN could be
in subcluster_old and subcluster_new at the same time and then decommission it
from subcluster_old.
In any case, the first version could prevent moving DNs out of the current
subcluster.
> RBF: Datanodes interacting with Routers
> ---------------------------------------
>
> Key: HDFS-13098
> URL: https://issues.apache.org/jira/browse/HDFS-13098
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Íñigo Goiri
> Priority: Major
>
> Datanodes talk to particular Namenodes. We could use the Router
> infrastructure for the Datanodes to register/heartbeating into them and the
> Routers would forward this to particular Namenodes. This would make the
> assignment of Datanodes to subclusters potentially more dynamic.
> The implementation would potentially include:
> * Router to implement part of DatanodeProtocol
> * Forwarding DN messages into Routers
> * Policies to assign datanodes to subclusters
> * Datanodes to make blockpool configuration dynamic
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
