[14/50] [abbrv] hadoop git commit: YARN-2801. Add documentation for node labels feature. Contributed by Wangda Tan and Naganarasimha G R.
YARN-2801. Add documentation for node labels feature. Contributed by Wangda Tan and Naganarasimha G R. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/faa38e1a Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/faa38e1a Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/faa38e1a Branch: refs/heads/YARN-3926 Commit: faa38e1aa49907254bf981662a8aeb5dc52e75e3 Parents: bf669b6 Author: Tsuyoshi OzawaAuthored: Tue Sep 1 17:54:14 2015 +0900 Committer: Tsuyoshi Ozawa Committed: Tue Sep 1 17:54:49 2015 +0900 -- hadoop-project/src/site/site.xml| 1 + hadoop-yarn-project/CHANGES.txt | 3 + .../src/site/markdown/NodeLabel.md | 140 +++ 3 files changed, 144 insertions(+) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/faa38e1a/hadoop-project/src/site/site.xml -- diff --git a/hadoop-project/src/site/site.xml b/hadoop-project/src/site/site.xml index ee0dfcd..272ea7b 100644 --- a/hadoop-project/src/site/site.xml +++ b/hadoop-project/src/site/site.xml @@ -121,6 +121,7 @@ + http://git-wip-us.apache.org/repos/asf/hadoop/blob/faa38e1a/hadoop-yarn-project/CHANGES.txt -- diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt index 999654d..e2b1307 100644 --- a/hadoop-yarn-project/CHANGES.txt +++ b/hadoop-yarn-project/CHANGES.txt @@ -827,6 +827,9 @@ Release 2.7.2 - UNRELEASED YARN-4092. Fixed UI redirection to print useful messages when both RMs are in standby mode. (Xuan Gong via jianhe) +YARN-2801. Add documentation for node labels feature. (Wangda Tan and Naganarasimha +G R via ozawa) + OPTIMIZATIONS BUG FIXES http://git-wip-us.apache.org/repos/asf/hadoop/blob/faa38e1a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeLabel.md -- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeLabel.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeLabel.md new file mode 100644 index 000..87019cd --- /dev/null +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeLabel.md @@ -0,0 +1,140 @@ + + +YARN Node Labels +=== + +# Overview +Node label is a way to group nodes with similar characteristics and applications can specify where to run. + +Now we only support node partition, which is: + +* One node can have only one node partition, so a cluster is partitioned to several disjoint sub-clusters by node partitions. By default, nodes belong to DEFAULT partition (partition="") +* User need to configure how much resources of each partition can be used by different queues. For more detail, please refer next section. +* There are two kinds of node partitions: +* Exclusive: containers will be allocated to nodes with exactly match node partition. (e.g. asking partition=âxâ will be allocated to node with partition=âxâ, asking DEFAULT partition will be allocated to DEFAULT partition nodes). +* Non-exclusive: if a partition is non-exclusive, it shares idle resource to container requesting DEFAULT partition. + +User can specify set of node labels which can be accessed by each queue, one application can only use subset of node labels that can be accessed by the queue which contains the application. + +# Features +The ```Node Labels``` supports the following features for now: + +* Partition cluster - each node can be assigned one label, so the cluster will be divided to several smaller disjoint partitions. +* ACL of node-labels on queues - user can set accessible node labels on each queue so only some nodes can only be accessed by specific queues. +* Specify percentage of resource of a partition which can be accessed by a queue - user can set percentage like: queue A can access 30% of resources on nodes with label=hbase. Such percentage setting will be consistent with existing resource manager +* Specify required Node Label in resource request, it will only be allocated when node has the same label. If no node label requirement specified, such Resource Request will only be allocated on nodes belong to DEFAULT partition. +* Operability +* Node labels and node labels mapping can be recovered across RM restart +* Update node labels - admin can update labels on nodes and labels on queues + when RM is running + +# Configuration +## Setting up ```ResourceManager``` to enable ```Node Labels```: + +Setup following properties in ```yarn-site.xml``` +
[13/50] [abbrv] hadoop git commit: YARN-2801. Add documentation for node labels feature. Contributed by Wangda Tan and Naganarasimha G R.
YARN-2801. Add documentation for node labels feature. Contributed by Wangda Tan and Naganarasimha G R. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/faa38e1a Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/faa38e1a Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/faa38e1a Branch: refs/heads/YARN-1197 Commit: faa38e1aa49907254bf981662a8aeb5dc52e75e3 Parents: bf669b6 Author: Tsuyoshi OzawaAuthored: Tue Sep 1 17:54:14 2015 +0900 Committer: Tsuyoshi Ozawa Committed: Tue Sep 1 17:54:49 2015 +0900 -- hadoop-project/src/site/site.xml| 1 + hadoop-yarn-project/CHANGES.txt | 3 + .../src/site/markdown/NodeLabel.md | 140 +++ 3 files changed, 144 insertions(+) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/faa38e1a/hadoop-project/src/site/site.xml -- diff --git a/hadoop-project/src/site/site.xml b/hadoop-project/src/site/site.xml index ee0dfcd..272ea7b 100644 --- a/hadoop-project/src/site/site.xml +++ b/hadoop-project/src/site/site.xml @@ -121,6 +121,7 @@ + http://git-wip-us.apache.org/repos/asf/hadoop/blob/faa38e1a/hadoop-yarn-project/CHANGES.txt -- diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt index 999654d..e2b1307 100644 --- a/hadoop-yarn-project/CHANGES.txt +++ b/hadoop-yarn-project/CHANGES.txt @@ -827,6 +827,9 @@ Release 2.7.2 - UNRELEASED YARN-4092. Fixed UI redirection to print useful messages when both RMs are in standby mode. (Xuan Gong via jianhe) +YARN-2801. Add documentation for node labels feature. (Wangda Tan and Naganarasimha +G R via ozawa) + OPTIMIZATIONS BUG FIXES http://git-wip-us.apache.org/repos/asf/hadoop/blob/faa38e1a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeLabel.md -- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeLabel.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeLabel.md new file mode 100644 index 000..87019cd --- /dev/null +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeLabel.md @@ -0,0 +1,140 @@ + + +YARN Node Labels +=== + +# Overview +Node label is a way to group nodes with similar characteristics and applications can specify where to run. + +Now we only support node partition, which is: + +* One node can have only one node partition, so a cluster is partitioned to several disjoint sub-clusters by node partitions. By default, nodes belong to DEFAULT partition (partition="") +* User need to configure how much resources of each partition can be used by different queues. For more detail, please refer next section. +* There are two kinds of node partitions: +* Exclusive: containers will be allocated to nodes with exactly match node partition. (e.g. asking partition=âxâ will be allocated to node with partition=âxâ, asking DEFAULT partition will be allocated to DEFAULT partition nodes). +* Non-exclusive: if a partition is non-exclusive, it shares idle resource to container requesting DEFAULT partition. + +User can specify set of node labels which can be accessed by each queue, one application can only use subset of node labels that can be accessed by the queue which contains the application. + +# Features +The ```Node Labels``` supports the following features for now: + +* Partition cluster - each node can be assigned one label, so the cluster will be divided to several smaller disjoint partitions. +* ACL of node-labels on queues - user can set accessible node labels on each queue so only some nodes can only be accessed by specific queues. +* Specify percentage of resource of a partition which can be accessed by a queue - user can set percentage like: queue A can access 30% of resources on nodes with label=hbase. Such percentage setting will be consistent with existing resource manager +* Specify required Node Label in resource request, it will only be allocated when node has the same label. If no node label requirement specified, such Resource Request will only be allocated on nodes belong to DEFAULT partition. +* Operability +* Node labels and node labels mapping can be recovered across RM restart +* Update node labels - admin can update labels on nodes and labels on queues + when RM is running + +# Configuration +## Setting up ```ResourceManager``` to enable ```Node Labels```: + +Setup following properties in ```yarn-site.xml``` +
hadoop git commit: YARN-2801. Add documentation for node labels feature. Contributed by Wangda Tan and Naganarasimha G R.
Repository: hadoop Updated Branches: refs/heads/branch-2.7 f44ed4f4b -> 1b5bf5800 YARN-2801. Add documentation for node labels feature. Contributed by Wangda Tan and Naganarasimha G R. (cherry picked from commit faa38e1aa49907254bf981662a8aeb5dc52e75e3) Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/1b5bf580 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/1b5bf580 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/1b5bf580 Branch: refs/heads/branch-2.7 Commit: 1b5bf58001d68e83534d4fc93959cc2c42540629 Parents: f44ed4f Author: Tsuyoshi OzawaAuthored: Tue Sep 1 17:54:14 2015 +0900 Committer: Tsuyoshi Ozawa Committed: Tue Sep 1 17:56:08 2015 +0900 -- hadoop-project/src/site/site.xml| 1 + hadoop-yarn-project/CHANGES.txt | 3 + .../src/site/markdown/NodeLabel.md | 140 +++ 3 files changed, 144 insertions(+) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/1b5bf580/hadoop-project/src/site/site.xml -- diff --git a/hadoop-project/src/site/site.xml b/hadoop-project/src/site/site.xml index 9fa7e4d..df93945 100644 --- a/hadoop-project/src/site/site.xml +++ b/hadoop-project/src/site/site.xml @@ -120,6 +120,7 @@ + http://git-wip-us.apache.org/repos/asf/hadoop/blob/1b5bf580/hadoop-yarn-project/CHANGES.txt -- diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt index e07c8ae..ee9b66a 100644 --- a/hadoop-yarn-project/CHANGES.txt +++ b/hadoop-yarn-project/CHANGES.txt @@ -20,6 +20,9 @@ Release 2.7.2 - UNRELEASED YARN-4092. Fixed UI redirection to print useful messages when both RMs are in standby mode. (Xuan Gong via jianhe) +YARN-2801. Add documentation for node labels feature. (Wangda Tan and Naganarasimha +G R via ozawa) + OPTIMIZATIONS BUG FIXES http://git-wip-us.apache.org/repos/asf/hadoop/blob/1b5bf580/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeLabel.md -- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeLabel.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeLabel.md new file mode 100644 index 000..87019cd --- /dev/null +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeLabel.md @@ -0,0 +1,140 @@ + + +YARN Node Labels +=== + +# Overview +Node label is a way to group nodes with similar characteristics and applications can specify where to run. + +Now we only support node partition, which is: + +* One node can have only one node partition, so a cluster is partitioned to several disjoint sub-clusters by node partitions. By default, nodes belong to DEFAULT partition (partition="") +* User need to configure how much resources of each partition can be used by different queues. For more detail, please refer next section. +* There are two kinds of node partitions: +* Exclusive: containers will be allocated to nodes with exactly match node partition. (e.g. asking partition=âxâ will be allocated to node with partition=âxâ, asking DEFAULT partition will be allocated to DEFAULT partition nodes). +* Non-exclusive: if a partition is non-exclusive, it shares idle resource to container requesting DEFAULT partition. + +User can specify set of node labels which can be accessed by each queue, one application can only use subset of node labels that can be accessed by the queue which contains the application. + +# Features +The ```Node Labels``` supports the following features for now: + +* Partition cluster - each node can be assigned one label, so the cluster will be divided to several smaller disjoint partitions. +* ACL of node-labels on queues - user can set accessible node labels on each queue so only some nodes can only be accessed by specific queues. +* Specify percentage of resource of a partition which can be accessed by a queue - user can set percentage like: queue A can access 30% of resources on nodes with label=hbase. Such percentage setting will be consistent with existing resource manager +* Specify required Node Label in resource request, it will only be allocated when node has the same label. If no node label requirement specified, such Resource Request will only be allocated on nodes belong to DEFAULT partition. +* Operability +* Node labels and node labels mapping can be recovered across RM restart +* Update node labels - admin can update labels on nodes and labels on queues + when RM is
hadoop git commit: YARN-2801. Add documentation for node labels feature. Contributed by Wangda Tan and Naganarasimha G R.
Repository: hadoop Updated Branches: refs/heads/trunk bf669b6d9 -> faa38e1aa YARN-2801. Add documentation for node labels feature. Contributed by Wangda Tan and Naganarasimha G R. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/faa38e1a Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/faa38e1a Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/faa38e1a Branch: refs/heads/trunk Commit: faa38e1aa49907254bf981662a8aeb5dc52e75e3 Parents: bf669b6 Author: Tsuyoshi OzawaAuthored: Tue Sep 1 17:54:14 2015 +0900 Committer: Tsuyoshi Ozawa Committed: Tue Sep 1 17:54:49 2015 +0900 -- hadoop-project/src/site/site.xml| 1 + hadoop-yarn-project/CHANGES.txt | 3 + .../src/site/markdown/NodeLabel.md | 140 +++ 3 files changed, 144 insertions(+) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/faa38e1a/hadoop-project/src/site/site.xml -- diff --git a/hadoop-project/src/site/site.xml b/hadoop-project/src/site/site.xml index ee0dfcd..272ea7b 100644 --- a/hadoop-project/src/site/site.xml +++ b/hadoop-project/src/site/site.xml @@ -121,6 +121,7 @@ + http://git-wip-us.apache.org/repos/asf/hadoop/blob/faa38e1a/hadoop-yarn-project/CHANGES.txt -- diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt index 999654d..e2b1307 100644 --- a/hadoop-yarn-project/CHANGES.txt +++ b/hadoop-yarn-project/CHANGES.txt @@ -827,6 +827,9 @@ Release 2.7.2 - UNRELEASED YARN-4092. Fixed UI redirection to print useful messages when both RMs are in standby mode. (Xuan Gong via jianhe) +YARN-2801. Add documentation for node labels feature. (Wangda Tan and Naganarasimha +G R via ozawa) + OPTIMIZATIONS BUG FIXES http://git-wip-us.apache.org/repos/asf/hadoop/blob/faa38e1a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeLabel.md -- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeLabel.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeLabel.md new file mode 100644 index 000..87019cd --- /dev/null +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeLabel.md @@ -0,0 +1,140 @@ + + +YARN Node Labels +=== + +# Overview +Node label is a way to group nodes with similar characteristics and applications can specify where to run. + +Now we only support node partition, which is: + +* One node can have only one node partition, so a cluster is partitioned to several disjoint sub-clusters by node partitions. By default, nodes belong to DEFAULT partition (partition="") +* User need to configure how much resources of each partition can be used by different queues. For more detail, please refer next section. +* There are two kinds of node partitions: +* Exclusive: containers will be allocated to nodes with exactly match node partition. (e.g. asking partition=âxâ will be allocated to node with partition=âxâ, asking DEFAULT partition will be allocated to DEFAULT partition nodes). +* Non-exclusive: if a partition is non-exclusive, it shares idle resource to container requesting DEFAULT partition. + +User can specify set of node labels which can be accessed by each queue, one application can only use subset of node labels that can be accessed by the queue which contains the application. + +# Features +The ```Node Labels``` supports the following features for now: + +* Partition cluster - each node can be assigned one label, so the cluster will be divided to several smaller disjoint partitions. +* ACL of node-labels on queues - user can set accessible node labels on each queue so only some nodes can only be accessed by specific queues. +* Specify percentage of resource of a partition which can be accessed by a queue - user can set percentage like: queue A can access 30% of resources on nodes with label=hbase. Such percentage setting will be consistent with existing resource manager +* Specify required Node Label in resource request, it will only be allocated when node has the same label. If no node label requirement specified, such Resource Request will only be allocated on nodes belong to DEFAULT partition. +* Operability +* Node labels and node labels mapping can be recovered across RM restart +* Update node labels - admin can update labels on nodes and labels on queues + when RM is running + +# Configuration +## Setting up ```ResourceManager``` to enable
hadoop git commit: YARN-2801. Add documentation for node labels feature. Contributed by Wangda Tan and Naganarasimha G R.
Repository: hadoop Updated Branches: refs/heads/branch-2 855e0f8b0 -> d3d65f62b YARN-2801. Add documentation for node labels feature. Contributed by Wangda Tan and Naganarasimha G R. (cherry picked from commit faa38e1aa49907254bf981662a8aeb5dc52e75e3) Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/d3d65f62 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/d3d65f62 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/d3d65f62 Branch: refs/heads/branch-2 Commit: d3d65f62b39dfe3f4d0efa47e5129e2aa3033d5e Parents: 855e0f8 Author: Tsuyoshi OzawaAuthored: Tue Sep 1 17:54:14 2015 +0900 Committer: Tsuyoshi Ozawa Committed: Tue Sep 1 17:55:53 2015 +0900 -- hadoop-project/src/site/site.xml| 1 + hadoop-yarn-project/CHANGES.txt | 3 + .../src/site/markdown/NodeLabel.md | 140 +++ 3 files changed, 144 insertions(+) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/d3d65f62/hadoop-project/src/site/site.xml -- diff --git a/hadoop-project/src/site/site.xml b/hadoop-project/src/site/site.xml index 0e2ab2e..9b888de 100644 --- a/hadoop-project/src/site/site.xml +++ b/hadoop-project/src/site/site.xml @@ -120,6 +120,7 @@ + http://git-wip-us.apache.org/repos/asf/hadoop/blob/d3d65f62/hadoop-yarn-project/CHANGES.txt -- diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt index c19bd7b..f8c6c96 100644 --- a/hadoop-yarn-project/CHANGES.txt +++ b/hadoop-yarn-project/CHANGES.txt @@ -775,6 +775,9 @@ Release 2.7.2 - UNRELEASED YARN-4092. Fixed UI redirection to print useful messages when both RMs are in standby mode. (Xuan Gong via jianhe) +YARN-2801. Add documentation for node labels feature. (Wangda Tan and Naganarasimha +G R via ozawa) + OPTIMIZATIONS BUG FIXES http://git-wip-us.apache.org/repos/asf/hadoop/blob/d3d65f62/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeLabel.md -- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeLabel.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeLabel.md new file mode 100644 index 000..87019cd --- /dev/null +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeLabel.md @@ -0,0 +1,140 @@ + + +YARN Node Labels +=== + +# Overview +Node label is a way to group nodes with similar characteristics and applications can specify where to run. + +Now we only support node partition, which is: + +* One node can have only one node partition, so a cluster is partitioned to several disjoint sub-clusters by node partitions. By default, nodes belong to DEFAULT partition (partition="") +* User need to configure how much resources of each partition can be used by different queues. For more detail, please refer next section. +* There are two kinds of node partitions: +* Exclusive: containers will be allocated to nodes with exactly match node partition. (e.g. asking partition=âxâ will be allocated to node with partition=âxâ, asking DEFAULT partition will be allocated to DEFAULT partition nodes). +* Non-exclusive: if a partition is non-exclusive, it shares idle resource to container requesting DEFAULT partition. + +User can specify set of node labels which can be accessed by each queue, one application can only use subset of node labels that can be accessed by the queue which contains the application. + +# Features +The ```Node Labels``` supports the following features for now: + +* Partition cluster - each node can be assigned one label, so the cluster will be divided to several smaller disjoint partitions. +* ACL of node-labels on queues - user can set accessible node labels on each queue so only some nodes can only be accessed by specific queues. +* Specify percentage of resource of a partition which can be accessed by a queue - user can set percentage like: queue A can access 30% of resources on nodes with label=hbase. Such percentage setting will be consistent with existing resource manager +* Specify required Node Label in resource request, it will only be allocated when node has the same label. If no node label requirement specified, such Resource Request will only be allocated on nodes belong to DEFAULT partition. +* Operability +* Node labels and node labels mapping can be recovered across RM restart +* Update node labels - admin can update labels on nodes and labels on queues + when RM is