[jira] [Commented] (YARN-4266) Allow users to enter containers as UID:GID pair instead of by username
[ https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16142956#comment-16142956 ] Eric Yang commented on YARN-4266: - [~dan...@cloudera.com] LDAP group mapping contains a number of limitations, like nested OU in AD, or non-standardized LDAP schema support. Application specific LDAP configuration usually have some limitations on supporting wide variety of LDAP servers. LDAP group mapping is a proof of concept module, and not recommended for production workload. Secure cluster requires to use Linux Container Executor, which requires the program to run as unix user that exists in LInux environment. Instead of letting people to run insecure cluster with proof of concept module. I would recommend to customer to support PAM for improving their security practices. Any self respected Linux administrator would not trust application to tap directly to enterprise LDAP server. > Allow users to enter containers as UID:GID pair instead of by username > -- > > Key: YARN-4266 > URL: https://issues.apache.org/jira/browse/YARN-4266 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: luhuichun > Attachments: YARN-4266.001.patch, YARN-4266.001.patch, > YARN-4266.002.patch, YARN-4266.003.patch, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf, > YARN-4266-branch-2.8.001.patch > > > Docker provides a mechanism (the --user switch) that enables us to specify > the user the container processes should run as. We use this mechanism today > when launching docker containers . In non-secure mode, we run the docker > container based on > `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in > secure mode, as the submitting user. However, this mechanism breaks down with > a large number of 'pre-created' images which don't necessarily have the users > available within the image. Examples of such images include shared images > that need to be used by multiple users. We need a way in which we can allow a > pre-defined set of users to run containers based on existing images, without > using the --user switch. There are some implications of disabling this user > squashing that we'll need to work through : log aggregation, artifact > deletion etc., -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6686) Support for adding and removing queue mappings
[ https://issues.apache.org/jira/browse/YARN-6686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16142952#comment-16142952 ] Esther Zheng commented on YARN-6686: [~jhung] One of my concerns regarding the current proposal is the order of the params. Here is an example: Initial value: {code:java} u:user1:queue1,u:%user:%user {code} {code:java} addParams: u:user1:queue2 removeParams: u:user1:queue1 {code} Actual value: {code:java} u:%user:%user,u:user1:queue2 {code} The current implementation adds the params to the end of the existing mappings. Note that the queue-mapping configuration is evaluated from left to right. In this example, the user 'user1' will be mapped to the queue 'user1' because 'u:%user:%user' is listed before 'u:user1:queue2'. However, the administrator may actually want to map 'user1' to 'queue2' (see below), i.e., 'u:user1:queue2' is placed before 'u:%user:%user'. Expected value: {code:java} u:user1:queue2,u:%user:%user {code} To achieve the expected value, the administrator may use: {code:java} addParams: u:user1:queue2,u:%user:%user removeParams: u:user1:queue1,u:%user:%user {code} Such usage may be inconvenient. Furthermore, if multiple global-updates occur simultaneously, the order still may not be the same as expected. Is this a valid concern? cc [~wangda] > Support for adding and removing queue mappings > -- > > Key: YARN-6686 > URL: https://issues.apache.org/jira/browse/YARN-6686 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Attachments: YARN-6686-YARN-5734.001.patch > > > Right now capacity scheduler uses UserGroupMappingPlacementRule to determine > queue mappings. This rule stores mappings in > {{yarn.scheduler.capacity.queue-mappings}}. For users with a large number of > mappings, adding or removing queue mappings becomes infeasible. > Need to come up with a way to add/remove individual mappings, for any/all > different configured placement rules. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6799) Remove the duplicated code in CGroupsHandlerImp.java
[ https://issues.apache.org/jira/browse/YARN-6799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16142853#comment-16142853 ] weiyuan commented on YARN-6799: --- Hi, [~yufeigu], thanks for filling this form. I would like to try this issue, so I assigned it to me directly. > Remove the duplicated code in CGroupsHandlerImp.java > > > Key: YARN-6799 > URL: https://issues.apache.org/jira/browse/YARN-6799 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Yufei Gu >Priority: Trivial > > The else clause in initializeCGroupController is not necessary. > {code} > public void initializeCGroupController(CGroupController controller) throws > ResourceHandlerException { > if (enableCGroupMount) { > // We have a controller that needs to be mounted > mountCGroupController(controller); > } else { > String controllerPath = getControllerPath(controller); > if (controllerPath == null) { > throw new ResourceHandlerException( > String.format("Controller %s not mounted." > + " You either need to mount it with %s" > + " or mount cgroups before launching Yarn", > controller.getName(), YarnConfiguration. > NM_LINUX_CONTAINER_CGROUPS_MOUNT)); > } > } > // We are working with a pre-mounted contoller > // Make sure that Yarn cgroup hierarchy path exists > initializePreMountedCGroupController(controller); > } > private void initializePreMountedCGroupController(CGroupController > controller) > throws ResourceHandlerException { > // Check permissions to cgroup hierarchy and > // create YARN cgroup if it does not exist, yet > String controllerPath = getControllerPath(controller); > if (controllerPath == null) { > throw new ResourceHandlerException( > String.format("Controller %s not mounted." > + " You either need to mount it with %s" > + " or mount cgroups before launching Yarn", > controller.getName(), YarnConfiguration. > NM_LINUX_CONTAINER_CGROUPS_MOUNT)); > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6799) Remove the duplicated code in CGroupsHandlerImp.java
[ https://issues.apache.org/jira/browse/YARN-6799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] weiyuan reassigned YARN-6799: - Assignee: weiyuan > Remove the duplicated code in CGroupsHandlerImp.java > > > Key: YARN-6799 > URL: https://issues.apache.org/jira/browse/YARN-6799 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Yufei Gu >Assignee: weiyuan >Priority: Trivial > > The else clause in initializeCGroupController is not necessary. > {code} > public void initializeCGroupController(CGroupController controller) throws > ResourceHandlerException { > if (enableCGroupMount) { > // We have a controller that needs to be mounted > mountCGroupController(controller); > } else { > String controllerPath = getControllerPath(controller); > if (controllerPath == null) { > throw new ResourceHandlerException( > String.format("Controller %s not mounted." > + " You either need to mount it with %s" > + " or mount cgroups before launching Yarn", > controller.getName(), YarnConfiguration. > NM_LINUX_CONTAINER_CGROUPS_MOUNT)); > } > } > // We are working with a pre-mounted contoller > // Make sure that Yarn cgroup hierarchy path exists > initializePreMountedCGroupController(controller); > } > private void initializePreMountedCGroupController(CGroupController > controller) > throws ResourceHandlerException { > // Check permissions to cgroup hierarchy and > // create YARN cgroup if it does not exist, yet > String controllerPath = getControllerPath(controller); > if (controllerPath == null) { > throw new ResourceHandlerException( > String.format("Controller %s not mounted." > + " You either need to mount it with %s" > + " or mount cgroups before launching Yarn", > controller.getName(), YarnConfiguration. > NM_LINUX_CONTAINER_CGROUPS_MOUNT)); > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6933) ResourceUtils.DISALLOWED_NAMES and ResourceUtils.checkMandatoryResources() are duplicating work
[ https://issues.apache.org/jira/browse/YARN-6933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16142851#comment-16142851 ] Daniel Templeton commented on YARN-6933: The message used for CPUs and the other memory check is: "Attempt to re-define mandatory resource 'memory-mb'. It can only be of type 'COUNTABLE' and have units 'Mi'." Can we use that same message here? > ResourceUtils.DISALLOWED_NAMES and ResourceUtils.checkMandatoryResources() > are duplicating work > --- > > Key: YARN-6933 > URL: https://issues.apache.org/jira/browse/YARN-6933 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: YARN-3926 >Reporter: Daniel Templeton >Assignee: Manikandan R > Labels: newbie++ > Attachments: YARN-6933-YARN-3926.001.patch, > YARN-6933-YARN-3926.002.patch, YARN-6933-YARN-3926.003.patch, > YARN-6933-YARN-3926.004.patch, YARN-6933-YARN-3926.005.patch, > YARN-6933-YARN-3926.006.patch, YARN-6933-YARN-3926.007.patch > > > Both are used to check that the mandatory resources were not redefined. Only > one check is needed. I would recommend dropping {{DISALLOWED_RESOURCES}}. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6475) Fix some long function checkstyle issues
[ https://issues.apache.org/jira/browse/YARN-6475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16142848#comment-16142848 ] Daniel Templeton commented on YARN-6475: Sure. [~miklos.szeg...@cloudera.com], wanna provide me a branch-2 patch when you get a chance? > Fix some long function checkstyle issues > > > Key: YARN-6475 > URL: https://issues.apache.org/jira/browse/YARN-6475 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Miklos Szegedi >Assignee: Soumabrata Chakraborty >Priority: Trivial > Labels: newbie > Fix For: 3.0.0-alpha4 > > Attachments: YARN-6475.001.patch > > > I am talking about these two: > {code} > ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java:441: > @Override:3: Method length is 176 lines (max allowed is 150). [MethodLength] > ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java:159: > @Override:3: Method length is 158 lines (max allowed is 150). [MethodLength] > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow users to enter containers as UID:GID pair instead of by username
[ https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16142846#comment-16142846 ] Daniel Templeton commented on YARN-4266: OK, I buy that, but if a site uses the LDAP mapping, I would assume it's because they want to avoid setting up PAM to use LDAP. Now that's not an option, and the LDAP mapping becomes kinda useless when Docker is in play. OR maybe I'm just misremembering how the mappings work. > Allow users to enter containers as UID:GID pair instead of by username > -- > > Key: YARN-4266 > URL: https://issues.apache.org/jira/browse/YARN-4266 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: luhuichun > Attachments: YARN-4266.001.patch, YARN-4266.001.patch, > YARN-4266.002.patch, YARN-4266.003.patch, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf, > YARN-4266-branch-2.8.001.patch > > > Docker provides a mechanism (the --user switch) that enables us to specify > the user the container processes should run as. We use this mechanism today > when launching docker containers . In non-secure mode, we run the docker > container based on > `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in > secure mode, as the submitting user. However, this mechanism breaks down with > a large number of 'pre-created' images which don't necessarily have the users > available within the image. Examples of such images include shared images > that need to be used by multiple users. We need a way in which we can allow a > pre-defined set of users to run containers based on existing images, without > using the --user switch. There are some implications of disabling this user > squashing that we'll need to work through : log aggregation, artifact > deletion etc., -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7091) Rename application to service in yarn-native-services
[ https://issues.apache.org/jira/browse/YARN-7091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16142680#comment-16142680 ] Hadoop QA commented on YARN-7091: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 48 new or modified test files. {color} | || || || || {color:brown} yarn-native-services Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 46s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 23s{color} | {color:green} yarn-native-services passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m 29s{color} | {color:green} yarn-native-services passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 35s{color} | {color:green} yarn-native-services passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m 43s{color} | {color:green} yarn-native-services passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project hadoop-assemblies hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 43s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core in yarn-native-services has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 22s{color} | {color:green} yarn-native-services passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 22s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 12m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m 57s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 42s{color} | {color:orange} root: The patch generated 168 new + 1021 unchanged - 194 fixed = 1189 total (was 1215) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 7m 3s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m 23s{color} | {color:green} The patch generated 0 new + 0 unchanged - 13 fixed = 0 total (was 13) {color} | | {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m 11s{color} | {color:green} There were no new shelldocs issues. {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 1s{color} | {color:red} The patch has 81 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 12s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project hadoop-assemblies hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s{color} | {color:green} hadoop-project in the patch passed. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s{color} | {color:green} hadoop-assemblies in the patch passed. {color} | |
[jira] [Commented] (YARN-7104) Improvement of Heatmap
[ https://issues.apache.org/jira/browse/YARN-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16142651#comment-16142651 ] Da Ding commented on YARN-7104: --- [~wangda], patch is ready. > Improvement of Heatmap > -- > > Key: YARN-7104 > URL: https://issues.apache.org/jira/browse/YARN-7104 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Da Ding >Assignee: Da Ding > Attachments: Screen Shot 2017-08-25 at 8.26.16 PM.png, > yarn-7104.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7075) [YARN-3368] Improvement of Web UI
[ https://issues.apache.org/jira/browse/YARN-7075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16142649#comment-16142649 ] Da Ding commented on YARN-7075: --- [~wangda], patch is ready. > [YARN-3368] Improvement of Web UI > -- > > Key: YARN-7075 > URL: https://issues.apache.org/jira/browse/YARN-7075 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Da Ding >Assignee: Da Ding > Attachments: Screen Shot 2017-08-22 at 8.36.07 PM.png, > yarn-7075.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6999) Add log about how to solve Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster
[ https://issues.apache.org/jira/browse/YARN-6999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16142639#comment-16142639 ] Hudson commented on YARN-6999: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12249 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/12249/]) YARN-6999. Add log about how to solve Error: Could not find or load main (gtcarrera9: rev 3bee7046c52bff681a5aecc956ba9be6c3543397) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java > Add log about how to solve Error: Could not find or load main class > org.apache.hadoop.mapreduce.v2.app.MRAppMaster > -- > > Key: YARN-6999 > URL: https://issues.apache.org/jira/browse/YARN-6999 > Project: Hadoop YARN > Issue Type: Improvement > Components: documentation, security >Affects Versions: 3.0.0-beta1 > Environment: All operating systems. >Reporter: Linlin Zhou >Assignee: Linlin Zhou >Priority: Minor > Labels: newbie > Fix For: 3.0.0-beta1, 2.9 > > Attachments: yarn-6999.002.patch, yarn-6999.003.patch, yarn-6999.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > According Setting up a Single Node Cluster > [https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html], > we would still failed to run the MapReduce job example. Due to a security > fix, yarn use user's environment variables to init, and user's environment > variable usually doesn't include MapReduce related settings. So we need to > add the related config in etc/hadoop/mapred-site.xml manually. Currently the > log only tells there is an Error: > Could not find or load main class > org.apache.hadoop.mapreduce.v2.app.MRAppMaster, without suggestion on how to > solve it. I want to add the useful suggestion in log. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org