Repository: hadoop Updated Branches: refs/heads/branch-2 ad1567c48 -> 675d36895
YARN-6757. Refactor the usage of yarn.nodemanager.linux-container-executor.cgroups.mount-path (Contributed by Miklos Szegedi via Daniel Templeton) (cherry picked from commit 47b145b9b4e81d781891abce8a6638f0b436acc4) Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/675d3689 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/675d3689 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/675d3689 Branch: refs/heads/branch-2 Commit: 675d36895063e40362e247064206508ca712e3a4 Parents: ad1567c Author: Daniel Templeton <templ...@apache.org> Authored: Tue Aug 8 10:33:26 2017 -0700 Committer: Daniel Templeton <templ...@apache.org> Committed: Tue Aug 8 11:12:34 2017 -0700 ---------------------------------------------------------------------- .../src/main/resources/yarn-default.xml | 43 ++++++++++----- .../linux/resources/CGroupsHandler.java | 15 +++++ .../linux/resources/CGroupsHandlerImpl.java | 26 +++++---- .../linux/resources/ResourceHandlerModule.java | 58 ++++++++++++++++++-- .../util/CgroupsLCEResourcesHandler.java | 53 ++++++++++++------ .../linux/resources/TestCGroupsHandlerImpl.java | 27 ++++++++- .../util/TestCgroupsLCEResourcesHandler.java | 31 +++++++++++ .../src/site/markdown/GracefulDecommission.md | 12 ++-- .../src/site/markdown/NodeManagerCgroups.md | 17 +++++- .../site/markdown/WritingYarnApplications.md | 4 +- .../src/site/markdown/registry/yarn-registry.md | 14 ++--- 11 files changed, 237 insertions(+), 63 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/hadoop/blob/675d3689/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml index 826ded7..a3b8f2c 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml @@ -134,7 +134,7 @@ <property> <description> - This configures the HTTP endpoint for Yarn Daemons.The following + This configures the HTTP endpoint for YARN Daemons.The following values are supported: - HTTP_ONLY : Service is provided only on http - HTTPS_ONLY : Service is provided only on https @@ -1036,14 +1036,14 @@ DeletionService will delete the application's localized file directory and log directory. - To diagnose Yarn application problems, set this property's value large + To diagnose YARN application problems, set this property's value large enough (for example, to 600 = 10 minutes) to permit examination of these directories. After changing the property's value, you must restart the nodemanager in order for it to have an effect. - The roots of Yarn applications' work directories is configurable with + The roots of YARN applications' work directories is configurable with the yarn.nodemanager.local-dirs property (see below), and the roots - of the Yarn applications' log directories is configurable with the + of the YARN applications' log directories is configurable with the yarn.nodemanager.log-dirs property (see also below). </description> <name>yarn.nodemanager.delete.debug-delay-sec</name> @@ -1476,28 +1476,45 @@ <property> <description>The cgroups hierarchy under which to place YARN proccesses (cannot contain commas). If yarn.nodemanager.linux-container-executor.cgroups.mount is false - (that is, if cgroups have been pre-configured) and the Yarn user has write + (that is, if cgroups have been pre-configured) and the YARN user has write access to the parent directory, then the directory will be created. - If the directory already exists, the administrator has to give Yarn + If the directory already exists, the administrator has to give YARN write permissions to it recursively. - Only used when the LCE resources handler is set to the CgroupsLCEResourcesHandler.</description> + This property only applies when the LCE resources handler is set to + CgroupsLCEResourcesHandler.</description> <name>yarn.nodemanager.linux-container-executor.cgroups.hierarchy</name> <value>/hadoop-yarn</value> </property> <property> <description>Whether the LCE should attempt to mount cgroups if not found. - Only used when the LCE resources handler is set to the CgroupsLCEResourcesHandler.</description> + This property only applies when the LCE resources handler is set to + CgroupsLCEResourcesHandler. + </description> <name>yarn.nodemanager.linux-container-executor.cgroups.mount</name> <value>false</value> </property> <property> - <description>Where the LCE should attempt to mount cgroups if not found. Common locations - include /sys/fs/cgroup and /cgroup; the default location can vary depending on the Linux - distribution in use. This path must exist before the NodeManager is launched. - Only used when the LCE resources handler is set to the CgroupsLCEResourcesHandler, and - yarn.nodemanager.linux-container-executor.cgroups.mount is true.</description> + <description>This property sets the path from which YARN will read the + CGroups configuration. YARN has built-in functionality to discover the + system CGroup mount paths, so use this property only if YARN's automatic + mount path discovery does not work. + + The path specified by this property must exist before the NodeManager is + launched. + If yarn.nodemanager.linux-container-executor.cgroups.mount is set to true, + YARN will first try to mount the CGroups at the specified path before + reading them. + If yarn.nodemanager.linux-container-executor.cgroups.mount is set to + false, YARN will read the CGroups at the specified path. + If this property is empty, YARN tries to detect the CGroups location. + + Please refer to NodeManagerCgroups.html in the documentation for further + details. + This property only applies when the LCE resources handler is set to + CgroupsLCEResourcesHandler. + </description> <name>yarn.nodemanager.linux-container-executor.cgroups.mount-path</name> </property> http://git-wip-us.apache.org/repos/asf/hadoop/blob/675d3689/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandler.java ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandler.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandler.java index 8fc35a8..82bd366 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandler.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandler.java @@ -23,6 +23,9 @@ package org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resourc import org.apache.hadoop.classification.InterfaceAudience; import org.apache.hadoop.classification.InterfaceStability; +import java.util.HashSet; +import java.util.Set; + /** * Provides CGroups functionality. Implementations are expected to be * thread-safe @@ -54,6 +57,18 @@ public interface CGroupsHandler { String getName() { return name; } + + /** + * Get the list of valid cgroup names. + * @return The set of cgroup name strings + */ + public static Set<String> getValidCGroups() { + HashSet<String> validCgroups = new HashSet<>(); + for (CGroupController controller : CGroupController.values()) { + validCgroups.add(controller.getName()); + } + return validCgroups; + } } String CGROUP_FILE_TASKS = "tasks"; http://git-wip-us.apache.org/repos/asf/hadoop/blob/675d3689/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandlerImpl.java ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandlerImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandlerImpl.java index 85b01cd..9fd20eb 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandlerImpl.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandlerImpl.java @@ -83,7 +83,7 @@ class CGroupsHandlerImpl implements CGroupsHandler { * @param mtab mount file location * @throws ResourceHandlerException if initialization failed */ - public CGroupsHandlerImpl(Configuration conf, PrivilegedOperationExecutor + CGroupsHandlerImpl(Configuration conf, PrivilegedOperationExecutor privilegedOperationExecutor, String mtab) throws ResourceHandlerException { this.cGroupPrefix = conf.get(YarnConfiguration. @@ -115,7 +115,7 @@ class CGroupsHandlerImpl implements CGroupsHandler { * PrivilegedContainerOperations * @throws ResourceHandlerException if initialization failed */ - public CGroupsHandlerImpl(Configuration conf, PrivilegedOperationExecutor + CGroupsHandlerImpl(Configuration conf, PrivilegedOperationExecutor privilegedOperationExecutor) throws ResourceHandlerException { this(conf, privilegedOperationExecutor, MTAB_FILE); } @@ -142,11 +142,18 @@ class CGroupsHandlerImpl implements CGroupsHandler { // the same hierarchy will be mounted at each mount point with the same // subsystem set. - Map<String, Set<String>> newMtab; + Map<String, Set<String>> newMtab = null; Map<CGroupController, String> cPaths; try { - // parse mtab - newMtab = parseMtab(mtabFile); + if (this.cGroupMountPath != null && !this.enableCGroupMount) { + newMtab = ResourceHandlerModule. + parseConfiguredCGroupPath(this.cGroupMountPath); + } + + if (newMtab == null) { + // parse mtab + newMtab = parseMtab(mtabFile); + } // find cgroup controller paths cPaths = initializeControllerPathsFromMtab(newMtab); @@ -203,10 +210,8 @@ class CGroupsHandlerImpl implements CGroupsHandler { throws IOException { Map<String, Set<String>> ret = new HashMap<>(); BufferedReader in = null; - HashSet<String> validCgroups = new HashSet<>(); - for (CGroupController controller : CGroupController.values()) { - validCgroups.add(controller.getName()); - } + Set<String> validCgroups = + CGroupsHandler.CGroupController.getValidCGroups(); try { FileInputStream fis = new FileInputStream(new File(mtab)); @@ -487,7 +492,8 @@ class CGroupsHandlerImpl implements CGroupsHandler { try (BufferedReader inl = new BufferedReader(new InputStreamReader(new FileInputStream(cgf + "/tasks"), "UTF-8"))) { - if ((str = inl.readLine()) != null) { + str = inl.readLine(); + if (str != null) { LOG.debug("First line in cgroup tasks file: " + cgf + " " + str); } } catch (IOException e) { http://git-wip-us.apache.org/repos/asf/hadoop/blob/675d3689/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerModule.java ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerModule.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerModule.java index 7fc04bd..4d137f0 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerModule.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerModule.java @@ -31,6 +31,13 @@ import org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileg import org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler; import org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler; +import java.io.File; +import java.io.IOException; +import java.util.Set; +import java.util.HashSet; +import java.util.Map; +import java.util.HashMap; +import java.util.Arrays; import java.util.ArrayList; import java.util.List; @@ -113,8 +120,8 @@ public class ResourceHandlerModule { } private static TrafficControlBandwidthHandlerImpl - getTrafficControlBandwidthHandler(Configuration conf) - throws ResourceHandlerException { + getTrafficControlBandwidthHandler(Configuration conf) + throws ResourceHandlerException { if (conf.getBoolean(YarnConfiguration.NM_NETWORK_RESOURCE_ENABLED, YarnConfiguration.DEFAULT_NM_NETWORK_RESOURCE_ENABLED)) { if (trafficControlBandwidthHandler == null) { @@ -137,8 +144,8 @@ public class ResourceHandlerModule { } public static OutboundBandwidthResourceHandler - getOutboundBandwidthResourceHandler(Configuration conf) - throws ResourceHandlerException { + getOutboundBandwidthResourceHandler(Configuration conf) + throws ResourceHandlerException { return getTrafficControlBandwidthHandler(conf); } @@ -176,7 +183,7 @@ public class ResourceHandlerModule { } private static CGroupsMemoryResourceHandlerImpl - getCgroupsMemoryResourceHandler( + getCgroupsMemoryResourceHandler( Configuration conf) throws ResourceHandlerException { if (cGroupsMemoryResourceHandler == null) { synchronized (MemoryResourceHandler.class) { @@ -229,4 +236,45 @@ public class ResourceHandlerModule { static void nullifyResourceHandlerChain() throws ResourceHandlerException { resourceHandlerChain = null; } + + /** + * If a cgroup mount directory is specified, it returns cgroup directories + * with valid names. + * The requirement is that each hierarchy has to be named with the comma + * separated names of subsystems supported. + * For example: /sys/fs/cgroup/cpu,cpuacct + * @param cgroupMountPath Root cgroup mount path (/sys/fs/cgroup in the + * example above) + * @return A path to cgroup subsystem set mapping in the same format as + * {@link CGroupsHandlerImpl#parseMtab(String)} + * @throws IOException if the specified directory cannot be listed + */ + public static Map<String, Set<String>> parseConfiguredCGroupPath( + String cgroupMountPath) throws IOException { + File cgroupDir = new File(cgroupMountPath); + File[] list = cgroupDir.listFiles(); + if (list == null) { + throw new IOException("Empty cgroup mount directory specified: " + + cgroupMountPath); + } + + Map<String, Set<String>> pathSubsystemMappings = new HashMap<>(); + Set<String> validCGroups = + CGroupsHandler.CGroupController.getValidCGroups(); + for (File candidate: list) { + Set<String> cgroupList = + new HashSet<>(Arrays.asList(candidate.getName().split(","))); + // Collect the valid subsystem names + cgroupList.retainAll(validCGroups); + if (!cgroupList.isEmpty()) { + if (candidate.isDirectory() && candidate.canWrite()) { + pathSubsystemMappings.put(candidate.getAbsolutePath(), cgroupList); + } else { + LOG.warn("The following cgroup is not a directory or it is not" + + " writable" + candidate.getAbsolutePath()); + } + } + } + return pathSubsystemMappings; + } } http://git-wip-us.apache.org/repos/asf/hadoop/blob/675d3689/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java index bca4fdc..7a89285 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java @@ -27,6 +27,7 @@ import java.io.InputStreamReader; import java.io.OutputStreamWriter; import java.io.PrintWriter; import java.io.Writer; +import java.util.Arrays; import java.util.ArrayList; import java.util.Collections; import java.util.HashMap; @@ -39,7 +40,6 @@ import java.util.regex.Pattern; import com.google.common.annotations.VisibleForTesting; -import com.google.common.collect.Sets; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; @@ -51,6 +51,8 @@ import org.apache.hadoop.yarn.conf.YarnConfiguration; import org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor; import org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperation; import org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupsCpuResourceHandlerImpl; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupsHandler; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.ResourceHandlerModule; import org.apache.hadoop.yarn.util.Clock; import org.apache.hadoop.yarn.util.ResourceCalculatorPlugin; import org.apache.hadoop.yarn.util.SystemClock; @@ -87,11 +89,11 @@ public class CgroupsLCEResourcesHandler implements LCEResourcesHandler { private long deleteCgroupTimeout; private long deleteCgroupDelay; - // package private for testing purposes + @VisibleForTesting Clock clock; private float yarnProcessors; - int nodeVCores; + private int nodeVCores; public CgroupsLCEResourcesHandler() { this.controllerPaths = new HashMap<String, String>(); @@ -132,8 +134,10 @@ public class CgroupsLCEResourcesHandler implements LCEResourcesHandler { this.strictResourceUsageMode = conf .getBoolean( - YarnConfiguration.NM_LINUX_CONTAINER_CGROUPS_STRICT_RESOURCE_USAGE, - YarnConfiguration.DEFAULT_NM_LINUX_CONTAINER_CGROUPS_STRICT_RESOURCE_USAGE); + YarnConfiguration + .NM_LINUX_CONTAINER_CGROUPS_STRICT_RESOURCE_USAGE, + YarnConfiguration + .DEFAULT_NM_LINUX_CONTAINER_CGROUPS_STRICT_RESOURCE_USAGE); int len = cgroupPrefix.length(); if (cgroupPrefix.charAt(len - 1) == '/') { @@ -169,8 +173,10 @@ public class CgroupsLCEResourcesHandler implements LCEResourcesHandler { if (systemProcessors != (int) yarnProcessors) { LOG.info("YARN containers restricted to " + yarnProcessors + " cores"); int[] limits = getOverallLimits(yarnProcessors); - updateCgroup(CONTROLLER_CPU, "", CPU_PERIOD_US, String.valueOf(limits[0])); - updateCgroup(CONTROLLER_CPU, "", CPU_QUOTA_US, String.valueOf(limits[1])); + updateCgroup(CONTROLLER_CPU, "", CPU_PERIOD_US, + String.valueOf(limits[0])); + updateCgroup(CONTROLLER_CPU, "", CPU_QUOTA_US, + String.valueOf(limits[1])); } else if (CGroupsCpuResourceHandlerImpl.cpuLimitsExist( pathForCgroup(CONTROLLER_CPU, ""))) { LOG.info("Removing CPU constraints for YARN containers."); @@ -178,8 +184,8 @@ public class CgroupsLCEResourcesHandler implements LCEResourcesHandler { } } - int[] getOverallLimits(float yarnProcessors) { - return CGroupsCpuResourceHandlerImpl.getOverallLimits(yarnProcessors); + int[] getOverallLimits(float yarnProcessorsArg) { + return CGroupsCpuResourceHandlerImpl.getOverallLimits(yarnProcessorsArg); } @@ -204,7 +210,7 @@ public class CgroupsLCEResourcesHandler implements LCEResourcesHandler { LOG.debug("createCgroup: " + path); } - if (! new File(path).mkdir()) { + if (!new File(path).mkdir()) { throw new IOException("Failed to create cgroup at " + path); } } @@ -251,7 +257,8 @@ public class CgroupsLCEResourcesHandler implements LCEResourcesHandler { try (BufferedReader inl = new BufferedReader(new InputStreamReader(new FileInputStream(cgf + "/tasks"), "UTF-8"))) { - if ((str = inl.readLine()) != null) { + str = inl.readLine(); + if (str != null) { LOG.debug("First line in cgroup tasks file: " + cgf + " " + str); } } catch (IOException e) { @@ -337,9 +344,9 @@ public class CgroupsLCEResourcesHandler implements LCEResourcesHandler { (containerVCores * yarnProcessors) / (float) nodeVCores; int[] limits = getOverallLimits(containerCPU); updateCgroup(CONTROLLER_CPU, containerName, CPU_PERIOD_US, - String.valueOf(limits[0])); + String.valueOf(limits[0])); updateCgroup(CONTROLLER_CPU, containerName, CPU_QUOTA_US, - String.valueOf(limits[1])); + String.valueOf(limits[1])); } } } @@ -400,6 +407,8 @@ public class CgroupsLCEResourcesHandler implements LCEResourcesHandler { private Map<String, Set<String>> parseMtab() throws IOException { Map<String, Set<String>> ret = new HashMap<String, Set<String>>(); BufferedReader in = null; + Set<String> validCgroups = + CGroupsHandler.CGroupController.getValidCGroups(); try { FileInputStream fis = new FileInputStream(new File(getMtabFileName())); @@ -415,8 +424,11 @@ public class CgroupsLCEResourcesHandler implements LCEResourcesHandler { String options = m.group(3); if (type.equals(CGROUPS_FSTYPE)) { - HashSet<String> value = Sets.newHashSet(options.split(",")); - ret.put(path, value); + Set<String> cgroupList = + new HashSet<>(Arrays.asList(options.split(","))); + // Collect the valid subsystem names + cgroupList.retainAll(validCgroups); + ret.put(path, cgroupList); } } } @@ -448,7 +460,16 @@ public class CgroupsLCEResourcesHandler implements LCEResourcesHandler { private void initializeControllerPaths() throws IOException { String controllerPath; - Map<String, Set<String>> parsedMtab = parseMtab(); + Map<String, Set<String>> parsedMtab = null; + + if (this.cgroupMountPath != null && !this.cgroupMount) { + parsedMtab = ResourceHandlerModule. + parseConfiguredCGroupPath(this.cgroupMountPath); + } + + if (parsedMtab == null) { + parsedMtab = parseMtab(); + } // CPU http://git-wip-us.apache.org/repos/asf/hadoop/blob/675d3689/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsHandlerImpl.java ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsHandlerImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsHandlerImpl.java index 7a4d39f..ab989cf 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsHandlerImpl.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsHandlerImpl.java @@ -573,4 +573,29 @@ public class TestCGroupsHandlerImpl { new File(new File(newMountPoint, "cpu"), this.hierarchy); assertTrue("Yarn cgroup should exist", hierarchyFile.exists()); } -} + + + @Test + public void testManualCgroupSetting() throws ResourceHandlerException { + YarnConfiguration conf = new YarnConfiguration(); + conf.set(YarnConfiguration.NM_LINUX_CONTAINER_CGROUPS_MOUNT_PATH, tmpPath); + conf.set(YarnConfiguration.NM_LINUX_CONTAINER_CGROUPS_HIERARCHY, + "/hadoop-yarn"); + File cpu = new File(new File(tmpPath, "cpuacct,cpu"), "/hadoop-yarn"); + + try { + Assert.assertTrue("temp dir should be created", cpu.mkdirs()); + + CGroupsHandlerImpl cGroupsHandler = new CGroupsHandlerImpl(conf, null); + cGroupsHandler.initializeCGroupController( + CGroupsHandler.CGroupController.CPU); + + Assert.assertEquals("CPU CGRoup path was not set", cpu.getAbsolutePath(), + new File(cGroupsHandler.getPathForCGroup( + CGroupsHandler.CGroupController.CPU, "")).getAbsolutePath()); + + } finally { + FileUtils.deleteQuietly(cpu); + } + } +} \ No newline at end of file http://git-wip-us.apache.org/repos/asf/hadoop/blob/675d3689/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestCgroupsLCEResourcesHandler.java ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestCgroupsLCEResourcesHandler.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestCgroupsLCEResourcesHandler.java index 1ed8fd8..7d8704f 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestCgroupsLCEResourcesHandler.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestCgroupsLCEResourcesHandler.java @@ -41,6 +41,8 @@ import java.util.Scanner; import java.util.Set; import java.util.concurrent.CountDownLatch; +import static org.mockito.Mockito.when; + @Deprecated public class TestCgroupsLCEResourcesHandler { private static File cgroupDir = null; @@ -388,4 +390,33 @@ public class TestCgroupsLCEResourcesHandler { FileUtils.deleteQuietly(memory); } } + + @Test + public void testManualCgroupSetting() throws IOException { + CgroupsLCEResourcesHandler handler = new CgroupsLCEResourcesHandler(); + YarnConfiguration conf = new YarnConfiguration(); + conf.set(YarnConfiguration.NM_LINUX_CONTAINER_CGROUPS_MOUNT_PATH, + cgroupDir.getAbsolutePath()); + handler.setConf(conf); + File cpu = new File(new File(cgroupDir, "cpuacct,cpu"), "/hadoop-yarn"); + + try { + Assert.assertTrue("temp dir should be created", cpu.mkdirs()); + + final int numProcessors = 4; + ResourceCalculatorPlugin plugin = + Mockito.mock(ResourceCalculatorPlugin.class); + Mockito.doReturn(numProcessors).when(plugin).getNumProcessors(); + Mockito.doReturn(numProcessors).when(plugin).getNumCores(); + when(plugin.getNumProcessors()).thenReturn(8); + handler.init(null, plugin); + + Assert.assertEquals("CPU CGRoup path was not set", cpu.getParent(), + handler.getControllerPaths().get("cpu")); + + } finally { + FileUtils.deleteQuietly(cpu); + } + } + } http://git-wip-us.apache.org/repos/asf/hadoop/blob/675d3689/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/GracefulDecommission.md ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/GracefulDecommission.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/GracefulDecommission.md index 2acb3d2..2e83ca2 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/GracefulDecommission.md +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/GracefulDecommission.md @@ -13,7 +13,7 @@ --> -Graceful Decommission of Yarn Nodes +Graceful Decommission of YARN Nodes =============== * [Overview](#overview) @@ -29,19 +29,19 @@ Graceful Decommission of Yarn Nodes Overview -------- -Yarn is scalable very easily: any new NodeManager could join to the configured ResourceManager and start to execute jobs. But to achieve full elasticity we need a decommissioning process which helps to remove existing nodes and down-scale the cluster. +YARN is scalable very easily: any new NodeManager could join to the configured ResourceManager and start to execute jobs. But to achieve full elasticity we need a decommissioning process which helps to remove existing nodes and down-scale the cluster. -Yarn Nodes could be decommissioned NORMAL or GRACEFUL. +YARN Nodes could be decommissioned NORMAL or GRACEFUL. -Normal Decommission of Yarn Nodes means an immediate shutdown. +Normal Decommission of YARN Nodes means an immediate shutdown. -Graceful Decommission of Yarn Nodes is the mechanism to decommission NMs while minimize the impact to running applications. Once a node is in DECOMMISSIONING state, RM won't schedule new containers on it and will wait for running containers and applications to complete (or until decommissioning timeout exceeded) before transition the node into DECOMMISSIONED. +Graceful Decommission of YARN Nodes is the mechanism to decommission NMs while minimize the impact to running applications. Once a node is in DECOMMISSIONING state, RM won't schedule new containers on it and will wait for running containers and applications to complete (or until decommissioning timeout exceeded) before transition the node into DECOMMISSIONED. ## Quick start To do a normal decommissioning: -1. Start a Yarn cluster (with NodeManageres and ResourceManager) +1. Start a YARN cluster (with NodeManageres and ResourceManager) 2. Start a yarn job (for example with `yarn jar...` ) 3. Add `yarn.resourcemanager.nodes.exclude-path` property to your `yarn-site.xml` (Note: you don't need to restart the ResourceManager) 4. Create a text file (the location is defined in the previous step) with one line which contains the name of a selected NodeManager http://git-wip-us.apache.org/repos/asf/hadoop/blob/675d3689/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeManagerCgroups.md ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeManagerCgroups.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeManagerCgroups.md index 2704f10..d362801 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeManagerCgroups.md +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeManagerCgroups.md @@ -17,7 +17,7 @@ Using CGroups with YARN <!-- MACRO{toc|fromDepth=0|toDepth=3} --> -CGroups is a mechanism for aggregating/partitioning sets of tasks, and all their future children, into hierarchical groups with specialized behaviour. CGroups is a Linux kernel feature and was merged into kernel version 2.6.24. From a YARN perspective, this allows containers to be limited in their resource usage. A good example of this is CPU usage. Without CGroups, it becomes hard to limit container CPU usage. Currently, CGroups is only used for limiting CPU usage. +CGroups is a mechanism for aggregating/partitioning sets of tasks, and all their future children, into hierarchical groups with specialized behaviour. CGroups is a Linux kernel feature and was merged into kernel version 2.6.24. From a YARN perspective, this allows containers to be limited in their resource usage. A good example of this is CPU usage. Without CGroups, it becomes hard to limit container CPU usage. CGroups Configuration --------------------- @@ -30,9 +30,9 @@ The following settings are related to setting up CGroups. These need to be set i |:---- |:---- | | `yarn.nodemanager.container-executor.class` | This should be set to "org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor". CGroups is a Linux kernel feature and is exposed via the LinuxContainerExecutor. | | `yarn.nodemanager.linux-container-executor.resources-handler.class` | This should be set to "org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler". Using the LinuxContainerExecutor doesn't force you to use CGroups. If you wish to use CGroups, the resource-handler-class must be set to CGroupsLCEResourceHandler. | -| `yarn.nodemanager.linux-container-executor.cgroups.hierarchy` | The cgroups hierarchy under which to place YARN proccesses(cannot contain commas). If yarn.nodemanager.linux-container-executor.cgroups.mount is false (that is, if cgroups have been pre-configured) and the Yarn user has write access to the parent directory, then the directory will be created. If the directory already exists, the administrator has to give Yarn write permissions to it recursively. | +| `yarn.nodemanager.linux-container-executor.cgroups.hierarchy` | The cgroups hierarchy under which to place YARN proccesses(cannot contain commas). If yarn.nodemanager.linux-container-executor.cgroups.mount is false (that is, if cgroups have been pre-configured) and the YARN user has write access to the parent directory, then the directory will be created. If the directory already exists, the administrator has to give YARN write permissions to it recursively. | | `yarn.nodemanager.linux-container-executor.cgroups.mount` | Whether the LCE should attempt to mount cgroups if not found - can be true or false. | -| `yarn.nodemanager.linux-container-executor.cgroups.mount-path` | Where the LCE should attempt to mount cgroups if not found. Common locations include /sys/fs/cgroup and /cgroup; the default location can vary depending on the Linux distribution in use. This path must exist before the NodeManager is launched. Only used when the LCE resources handler is set to the CgroupsLCEResourcesHandler, and yarn.nodemanager.linux-container-executor.cgroups.mount is true. A point to note here is that the container-executor binary will try to mount the path specified + "/" + the subsystem. In our case, since we are trying to limit CPU the binary tries to mount the path specified + "/cpu" and that's the path it expects to exist. | +| `yarn.nodemanager.linux-container-executor.cgroups.mount-path` | Optional. Where CGroups are located. LCE will try to mount them here, if `yarn.nodemanager.linux-container-executor.cgroups.mount` is true. LCE will try to use CGroups from this location, if `yarn.nodemanager.linux-container-executor.cgroups.mount` is false. If specified, this path and its subdirectories (CGroup hierarchies) must exist and they should be readable and writable by YARN before the NodeManager is launched. See CGroups mount options below for details. | | `yarn.nodemanager.linux-container-executor.group` | The Unix group of the NodeManager. It should match the setting in "container-executor.cfg". This configuration is required for validating the secure access of the container-executor binary. | The following settings are related to limiting resource usage of YARN containers: @@ -42,6 +42,17 @@ The following settings are related to limiting resource usage of YARN containers | `yarn.nodemanager.resource.percentage-physical-cpu-limit` | This setting lets you limit the cpu usage of all YARN containers. It sets a hard upper limit on the cumulative CPU usage of the containers. For example, if set to 60, the combined CPU usage of all YARN containers will not exceed 60%. | | `yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage` | CGroups allows cpu usage limits to be hard or soft. When this setting is true, containers cannot use more CPU usage than allocated even if spare CPU is available. This ensures that containers can only use CPU that they were allocated. When set to false, containers can use spare CPU if available. It should be noted that irrespective of whether set to true or false, at no time can the combined CPU usage of all containers exceed the value specified in "yarn.nodemanager.resource.percentage-physical-cpu-limit". | +CGroups mount options +--------------------- + +YARN uses CGroups through a directory structure mounted into the file system by the kernel. There are three options to attach to CGroups. + +| Option | Description | +|:---- |:---- | +| Discover CGroups mounted already | This should be used on newer systems like RHEL7 or Ubuntu16 or if the administrator mounts CGroups before YARN starts. Set `yarn.nodemanager.linux-container-executor.cgroups.mount` to false and leave other settings set to their defaults. YARN will locate the mount points in `/proc/mounts`. Common locations include `/sys/fs/cgroup` and `/cgroup`. The default location can vary depending on the Linux distribution in use.| +| CGroups mounted by YARN | If the system does not have CGroups mounted or it is mounted to an inaccessible location then point `yarn.nodemanager.linux-container-executor.cgroups.mount-path` to an empty directory. Set `yarn.nodemanager.linux-container-executor.cgroups.mount` to true. A point to note here is that the container-executor binary will try to create and mount each subsystem as a subdirectory under this path. If `cpu` is already mounted somewhere with `cpuacct`, then the directory `cpu,cpuacct` will be created for the hierarchy.| +| CGroups mounted already or linked but not in `/proc/mounts` | If cgroups is accessible through lxcfs or simulated by another filesystem, then point `yarn.nodemanager.linux-container-executor.cgroups.mount-path` to your CGroups root directory. Set `yarn.nodemanager.linux-container-executor.cgroups.mount` to false. YARN tries to use this path first, before any CGroup mount point discovery. The path should have a subdirectory for each CGroup hierarchy named by the comma separated CGroup subsystems supported like `<path>/cpu,cpuacct`. Valid subsystem names are `cpu, cpuacct, cpuset, memory, net_cls, blkio, freezer, devices`.| + CGroups and security -------------------- http://git-wip-us.apache.org/repos/asf/hadoop/blob/675d3689/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/WritingYarnApplications.md ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/WritingYarnApplications.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/WritingYarnApplications.md index 07c3765..f1308d5 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/WritingYarnApplications.md +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/WritingYarnApplications.md @@ -56,7 +56,7 @@ Following are the important interfaces: * Under very rare circumstances, programmer may want to directly use the 3 protocols to implement an application. However, note that *such behaviors are no longer encouraged for general use cases*. -Writing a Simple Yarn Application +Writing a Simple YARN Application --------------------------------- ### Writing a simple Client @@ -574,4 +574,4 @@ Useful Links Sample Code ----------- -Yarn distributed shell: in `hadoop-yarn-applications-distributedshell` project after you set up your development environment. +YARN distributed shell: in `hadoop-yarn-applications-distributedshell` project after you set up your development environment. http://git-wip-us.apache.org/repos/asf/hadoop/blob/675d3689/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/registry/yarn-registry.md ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/registry/yarn-registry.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/registry/yarn-registry.md index f5055d9..4973862 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/registry/yarn-registry.md +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/registry/yarn-registry.md @@ -84,7 +84,7 @@ container ID. ## The binding problem Hadoop YARN allows applications to run on the Hadoop cluster. Some of these are -batch jobs or queries that can managed via Yarnâs existing API using its +batch jobs or queries that can managed via YARNâs existing API using its application ID. In addition YARN can deploy ong-lived services instances such a pool of Apache Tomcat web servers or an Apache HBase cluster. YARN will deploy them across the cluster depending on the individual each component requirements @@ -121,7 +121,7 @@ services accessible from within the Hadoop cluster /services/yarn /services/oozie -Yarn-deployed services belonging to individual users. +YARN-deployed services belonging to individual users. /users/joe/org-apache-hbase/demo1 /users/joe/org-apache-hbase/demo1/components/regionserver1 @@ -148,7 +148,7 @@ their application master, to which they heartbeat regularly. ## Unsupported Registration use cases: -1. A short-lived Yarn application is registered automatically in the registry, +1. A short-lived YARN application is registered automatically in the registry, including all its containers. and unregistered when the job terminates. Short-lived applications with many containers will place excessive load on a registry. All YARN applications will be given the option of registering, but it @@ -259,7 +259,7 @@ service since it supports many of the properties, We pick a part of the ZK namespace to be the root of the service registry ( default: `yarnRegistry`). On top this base implementation we build our registry service API and the -naming conventions that Yarn will use for its services. The registry will be +naming conventions that YARN will use for its services. The registry will be accessed by the registry API, not directly via ZK - ZK is just an implementation choice (although unlikely to change in the future). @@ -297,7 +297,7 @@ them. 6. Core services will be registered using the following convention: `/services/{servicename}` e.g. `/services/hdfs`. -7. Yarn services SHOULD be registered using the following convention: +7. YARN services SHOULD be registered using the following convention: /users/{username}/{serviceclass}/{instancename} @@ -823,8 +823,8 @@ The `RegistryPathStatus` class summarizes the contents of a node in the registry ## Security The registry will allow a service instance can only be registered under the -path where it has permissions. Yarn will create directories with appropriate -permissions for users where Yarn deployed services can be registered by a user. +path where it has permissions. YARN will create directories with appropriate +permissions for users where YARN deployed services can be registered by a user. of the user account of the service instance. The admin will also create directories (such as `/services`) with appropriate permissions (where core Hadoop services can register themselves. --------------------------------------------------------------------- To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
