[
https://issues.apache.org/jira/browse/HADOOP-7359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13061384#comment-13061384
]
Travis Crawford commented on HADOOP-7359:
-----------------------------------------
@steve - Great comments, thanks for taking a look. I agree the patch became
quite large and a safer approach is separating refactoring from functional
changes. I'll do that.
Regarding {{get[Excluded]Hosts()}} never failing, an explicit goal was no
change in today's behavior, unless you choose to plugin your own
{{HostsReader}}, and today's reader assumes the file is quick to read and
available. Hopefully anyone implementing a plugin uses automatic refresh, and
only triggers a refresh when the lists were successfully updated.
When manually calling {{hadoop dfsadmin -refreshNodes}} it may throw an
{{IOException}} indicating a failure to refresh, and plugins that choose to
automatically refresh should only do so after successfully updating.
> Pluggable interface for cluster membership
> ------------------------------------------
>
> Key: HADOOP-7359
> URL: https://issues.apache.org/jira/browse/HADOOP-7359
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Travis Crawford
> Attachments: HADOOP-7359.diff
>
>
> Currently Hadoop uses local files to determine cluster membership. With HDFS
> for example, dfs.hosts and dfs.hosts.exclude are used.
> To enable tighter integrations cluster membership should be an interface,
> with the current file-based functionality provided as the default
> implementation. The common case would be no functional change, however, sites
> could plug an alternative implementation in, such as pulling the machine
> lists from a machine database.
> DETAILS:
> Two machine lists, includes and excludes, are used to define cluster
> membership and state. HostsFileReader currently handles reading these lists
> from files, who's names are passed in by FSNamesystem for HDFS and JobTracker
> for MR.
> The proposed change is adding a HostsReader interface to common, and changing
> HostsFileReader to an abstract class that functions the same as today.
> Two new classes, DFSHostsFileReader and MRHostsFileReader, extend
> HostsFileReader and simply pass the appropriate file names in. These new
> classes are needed because config key names live outside common.
> Two new conf keys, defaulting to the file-based readers, would be added to
> choose a different hosts reader: dfs.namenode.hosts.reader.class
> mapreduce.jobtracker.hosts.reader.class
> Comments/suggestions? I have most of this written already but would love some
> feedback on the general idea before posting the diff.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
