[ https://issues.apache.org/jira/browse/YARN-8256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16467929#comment-16467929 ]
ASF GitHub Bot commented on YARN-8256: -------------------------------------- GitHub user functicons opened a pull request: https://github.com/apache/hadoop/pull/384 YARN-8256. Support pluggable node membership manager in YARN. [YARN-8256](https://issues.apache.org/jira/browse/YARN-8256): Support pluggable node membership manager in YARN. You can merge this pull request into a Git repository by running: $ git pull https://github.com/functicons/hadoop trunk Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop/pull/384.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #384 ---- commit 57a92fc8712d4fb4c6b88dc390baa2c5b30471ca Author: Dagang Wei <functicons@...> Date: 2018-05-07T20:41:59Z Support YARN pluggable node membership. ---- > Pluggable provider for node membership management > ------------------------------------------------- > > Key: YARN-8256 > URL: https://issues.apache.org/jira/browse/YARN-8256 > Project: Hadoop YARN > Issue Type: New Feature > Components: resourcemanager > Affects Versions: 2.8.3, 3.0.2 > Reporter: Dagang Wei > Priority: Major > Original Estimate: 24h > Remaining Estimate: 24h > > h1. Background > HDFS-7541 introduced a pluggable provider framework for node membership > management, which gives HDFS the flexibility to have different ways to manage > node membership for different needs. > [org.apache.hadoop.hdfs.server.blockmanagement.HostConfigManager|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HostConfigManager.java] > is the class which provides the abstraction. Currently, there are 2 > implementations in the HDFS codebase: > 1) > [org.apache.hadoop.hdfs.server.blockmanagement.HostFileManager|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HostFileManager.java] > which uses 2 config files which are defined by the properties dfs.hosts and > dfs.hosts.exclude. > 2) > [org.apache.hadoop.hdfs.server.blockmanagement.CombinedHostFileManager|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CombinedHostFileManager.java] > which uses a single JSON file defined by the property dfs.hosts. > dfs.namenode.hosts.provider.classname is the property determining which > implementation is used > h1. Problem > YARN should be consistent with HDFS in terms of pluggable provider for node > membership management. The absence of it makes YARN impossible to have other > config sources, e.g., ZooKeeper, database, other config file formats, etc. > h1. Proposed solution > [org.apache.hadoop.yarn.server.resourcemanager.NodesListManager|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java] > is the class for managing YARN node membership today. It uses > [HostsFileReader|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HostsFileReader.java] > to read config files specified by the property > yarn.resourcemanager.nodes.include-path for nodes to include and > yarn.resourcemanager.nodes.nodes.exclude-path for nodes to exclude. > The proposed solution is to > 1) introduce a new interface {color:#008000}HostsConfigManager{color} which > provides the abstraction for node membership management. Update > {color:#008000}NodeListManager{color} to depend on > {color:#008000}HostsConfigManager{color} instead of > {color:#008000}HostsFileReader{color}. Then create a wrapper class for > {color:#008000}HostsFileReader{color} which implements the interface. > 2) introduce a new config property > {color:#008000}yarn.resourcemanager.hosts-config.manager.class{color} for > specifying the implementation class. Set the default value to the wrapper > class of {color:#008000}HostsFileReader{color} for backward compatibility > between new code and old config. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org