Íñigo Goiri commented on HDFS-13098:

bq. Do these subclusters also dynamically generated? That is say, we have N 
datanodes, then these datanodes communicate with the Router. Based on the 
assignment policies, the Router decides which subcluster the datanode belong to.
We could add new subclusters (i.e., new namenodes) and then we can use 
consistent hashing to just move a small subset of the servers to the new 

bq. But if the machines are moving across subclusters, how we plan to do? 
Decommission in original subclsuter then add into new subcluster?
For now we would do this change of subclusters for servers that are empty: new 
ones or servers that get reimaged/decommissioned.
Once we support rebalancing, we could merge both things.
At the beginning, we can leverage regular HDFS federation and the DN could be 
in subcluster_old and subcluster_new at the same time and then decommission it 
from subcluster_old.
In any case, the first version could prevent moving DNs out of the current 

> RBF: Datanodes interacting with Routers
> ---------------------------------------
>                 Key: HDFS-13098
>                 URL: https://issues.apache.org/jira/browse/HDFS-13098
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Íñigo Goiri
>            Priority: Major
> Datanodes talk to particular Namenodes. We could use the Router 
> infrastructure for the Datanodes to register/heartbeating into them and the 
> Routers would forward this to particular Namenodes. This would make the 
> assignment of Datanodes to subclusters potentially more dynamic.
> The implementation would potentially include:
> * Router to implement part of DatanodeProtocol
> * Forwarding DN messages into Routers
> * Policies to assign datanodes to subclusters
> * Datanodes to make blockpool configuration dynamic

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

Reply via email to