Íñigo Goiri commented on HDFS-13098:

{quote}If I understand correctly, here the first hearbeat to Routers is to 
determine which subcluster this new DN will join. Afterwards, the DN will only 
talk to NNs in that subcluster. If this's correct, the first heartbeat to 
Routers is just like an "external" tool to generate the hdfs-site.xml, right?
My original idea was to do all the communications between the Router and the 
Namenode through the Routers. However, after discussing with [~curino], we 
realized that this might be a little heavy in most scenarios. I would still 
like to have an option to do everything through the Routers and another for 
just the registration.

In general this is like a tool to generate the {{hdfs-site.xml}} but it's a 
little more dynamic and easier to maintain for us as it would be in the same 
code repo as the Router (and the rest of Hadoop).
{quote}Do u have some example policies here, to determine the DN assignment? 
I'm thinking about data assignment policies, like allocating data to subcluter 
to balance subcluster workload, moving "warm" data to cheaper hardware 
subcluster, etc.
Currently, the policies we have are based on:
 * Number of subclusters we want
 * Consistent hashing (this allow us to distribute the nodes somewhat evenly 
and when the number of subclusters change, there is not much shuffling).
 * Maximum number of DNs we want per subcluster (we could extend that based on 
the number of files, block, total storage...)
 * Ad-hoc policies (machines from this rack will go to this subcluster)

{quote}Maybe I miss sth here, but do we need to relocate DNs from clusters to 
clusters quite often?
Right now, the only case where machines change is when we decide to add a new 
subcluster. However, the main use of this features for us is when new machines 
join the cluster.

Keep in mind that we could merge this with regular HDFS federation and have 
some DNs being part of two subclusters at the same time. This would make 
rebalancing faster if we support block hard linking for example.

In general, this would enable us to have a fully dynamic HDFS where we 
dynamically move DNs across subclusters. An example of this would be ideal for 
the case that [~chris.douglas] mentioned in the past where we could span a 
Namenode for a particular large application and add DNs dynamically.

> RBF: Datanodes interacting with Routers
> ---------------------------------------
>                 Key: HDFS-13098
>                 URL: https://issues.apache.org/jira/browse/HDFS-13098
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Íñigo Goiri
>            Priority: Major
> Datanodes talk to particular Namenodes. We could use the Router 
> infrastructure for the Datanodes to register/heartbeating into them and the 
> Routers would forward this to particular Namenodes. This would make the 
> assignment of Datanodes to subclusters potentially more dynamic.
> The implementation would potentially include:
> * Router to implement part of DatanodeProtocol
> * Forwarding DN messages into Routers
> * Policies to assign datanodes to subclusters
> * Datanodes to make blockpool configuration dynamic

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

Reply via email to