Pluggable interface for cluster membership
------------------------------------------
Key: HADOOP-7359
URL: https://issues.apache.org/jira/browse/HADOOP-7359
Project: Hadoop Common
Issue Type: Improvement
Reporter: Travis Crawford
Currently Hadoop uses local files to determine cluster membership. With HDFS
for example, dfs.hosts and dfs.hosts.exclude are used.
To enable tighter integrations cluster membership should be an interface, with
the current file-based functionality provided as the default implementation.
The common case would be no functional change, however, sites could plug an
alternative implementation in, such as pulling the machine lists from a machine
database.
DETAILS:
Two machine lists, includes and excludes, are used to define cluster membership
and state. HostsFileReader currently handles reading these lists from files,
who's names are passed in by FSNamesystem for HDFS and JobTracker for MR.
The proposed change is adding a HostsReader interface to common, and changing
HostsFileReader to an abstract class that functions the same as today.
Two new classes, DFSHostsFileReader and MRHostsFileReader, extend
HostsFileReader and simply pass the appropriate file names in. These new
classes are needed because config key names live outside common.
Two new conf keys, defaulting to the file-based readers, would be added to
choose a different hosts reader: dfs.namenode.hosts.reader.class
mapreduce.jobtracker.hosts.reader.class
Comments/suggestions? I have most of this written already but would love some
feedback on the general idea before posting the diff.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
