Abstract node to switch mapping into a topology service class used by namenode
and jobtracker
---------------------------------------------------------------------------------------------
Key: HADOOP-1985
URL: https://issues.apache.org/jira/browse/HADOOP-1985
Project: Hadoop
Issue Type: New Feature
Reporter: eric baldeschwieler
In order to implement switch locality in MapReduce, we need to have switch
location in both the namenode and job tracker. Currently the namenode asks the
data nodes for this info and they run a local script to answer this question.
In our environment and others that I know of there is no reason to push this to
each node. It is easier to maintain a centralized script that maps node DNS
names to switch strings.
I propose that we build a new class that caches known DNS name to switch
mappings and invokes a loadable class or a configurable system call to resolve
unknown DNS to switch mappings. We can then add this to the namenode to
support the current block to switch mapping needs and simplify the data nodes.
We can also add this same callout to the job tracker and then implement rack
locality logic there without needing to chane the filesystem API or the split
planning API.
Not only is this the least intrusive path to building racklocal MR I can ID, it
is also future compatible to future infrastructures that may derive topology on
the fly, etc, etc...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
