[
https://issues.apache.org/jira/browse/HDFS-13098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16349110#comment-16349110
]
Íñigo Goiri commented on HDFS-13098:
------------------------------------
{quote}If I understand correctly, here the first hearbeat to Routers is to
determine which subcluster this new DN will join. Afterwards, the DN will only
talk to NNs in that subcluster. If this's correct, the first heartbeat to
Routers is just like an "external" tool to generate the hdfs-site.xml, right?
{quote}
My original idea was to do all the communications between the Router and the
Namenode through the Routers. However, after discussing with [~curino], we
realized that this might be a little heavy in most scenarios. I would still
like to have an option to do everything through the Routers and another for
just the registration.
In general this is like a tool to generate the {{hdfs-site.xml}} but it's a
little more dynamic and easier to maintain for us as it would be in the same
code repo as the Router (and the rest of Hadoop).
{quote}Do u have some example policies here, to determine the DN assignment?
I'm thinking about data assignment policies, like allocating data to subcluter
to balance subcluster workload, moving "warm" data to cheaper hardware
subcluster, etc.
{quote}
Currently, the policies we have are based on:
* Number of subclusters we want
* Consistent hashing (this allow us to distribute the nodes somewhat evenly
and when the number of subclusters change, there is not much shuffling).
* Maximum number of DNs we want per subcluster (we could extend that based on
the number of files, block, total storage...)
* Ad-hoc policies (machines from this rack will go to this subcluster)
{quote}Maybe I miss sth here, but do we need to relocate DNs from clusters to
clusters quite often?
{quote}
Right now, the only case where machines change is when we decide to add a new
subcluster. However, the main use of this features for us is when new machines
join the cluster.
Keep in mind that we could merge this with regular HDFS federation and have
some DNs being part of two subclusters at the same time. This would make
rebalancing faster if we support block hard linking for example.
In general, this would enable us to have a fully dynamic HDFS where we
dynamically move DNs across subclusters. An example of this would be ideal for
the case that [~chris.douglas] mentioned in the past where we could span a
Namenode for a particular large application and add DNs dynamically.
> RBF: Datanodes interacting with Routers
> ---------------------------------------
>
> Key: HDFS-13098
> URL: https://issues.apache.org/jira/browse/HDFS-13098
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Íñigo Goiri
> Priority: Major
>
> Datanodes talk to particular Namenodes. We could use the Router
> infrastructure for the Datanodes to register/heartbeating into them and the
> Routers would forward this to particular Namenodes. This would make the
> assignment of Datanodes to subclusters potentially more dynamic.
> The implementation would potentially include:
> * Router to implement part of DatanodeProtocol
> * Forwarding DN messages into Routers
> * Policies to assign datanodes to subclusters
> * Datanodes to make blockpool configuration dynamic
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
