[
https://issues.apache.org/jira/browse/HADOOP-7359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Travis Crawford updated HADOOP-7359:
------------------------------------
Attachment: HADOOP-7359.diff
Attached patch contains changes for common, HDFS, and MR. I can break this into
3 patches if necessary (started a thread on common-dev about changes spanning
projects, post merge).
This patch adds automatic refreshing, enabled with the config keys:
{{dfs.namenode.hosts.reader.refresh.sec}}
{{mapreduce.jobtracker.hosts.reader.refresh.sec}}
By default refreshing is not enabled, so no functional change from today.
However, users could enable auto-refreshing, or plug their own {{HostsReader}}
implementation in.
> Pluggable interface for cluster membership
> ------------------------------------------
>
> Key: HADOOP-7359
> URL: https://issues.apache.org/jira/browse/HADOOP-7359
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Travis Crawford
> Attachments: HADOOP-7359.diff
>
>
> Currently Hadoop uses local files to determine cluster membership. With HDFS
> for example, dfs.hosts and dfs.hosts.exclude are used.
> To enable tighter integrations cluster membership should be an interface,
> with the current file-based functionality provided as the default
> implementation. The common case would be no functional change, however, sites
> could plug an alternative implementation in, such as pulling the machine
> lists from a machine database.
> DETAILS:
> Two machine lists, includes and excludes, are used to define cluster
> membership and state. HostsFileReader currently handles reading these lists
> from files, who's names are passed in by FSNamesystem for HDFS and JobTracker
> for MR.
> The proposed change is adding a HostsReader interface to common, and changing
> HostsFileReader to an abstract class that functions the same as today.
> Two new classes, DFSHostsFileReader and MRHostsFileReader, extend
> HostsFileReader and simply pass the appropriate file names in. These new
> classes are needed because config key names live outside common.
> Two new conf keys, defaulting to the file-based readers, would be added to
> choose a different hosts reader: dfs.namenode.hosts.reader.class
> mapreduce.jobtracker.hosts.reader.class
> Comments/suggestions? I have most of this written already but would love some
> feedback on the general idea before posting the diff.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
