[jira] [Commented] (YARN-5592) Add support for dynamic resource updates with multiple resource types
[ https://issues.apache.org/jira/browse/YARN-5592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344532#comment-16344532 ] Varun Vasudev commented on YARN-5592: - [~maniraj...@gmail.com] - this ticket was created to extend functionality added by YARN-291 to all resource types. > Add support for dynamic resource updates with multiple resource types > - > > Key: YARN-5592 > URL: https://issues.apache.org/jira/browse/YARN-5592 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Manikandan R >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-5592) Add support for dynamic resource updates with multiple resource types
[ https://issues.apache.org/jira/browse/YARN-5592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev reassigned YARN-5592: --- Assignee: Manikandan R (was: Varun Vasudev) > Add support for dynamic resource updates with multiple resource types > - > > Key: YARN-5592 > URL: https://issues.apache.org/jira/browse/YARN-5592 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Manikandan R >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-5590) Add support for increase and decrease of container resources with resource profiles
[ https://issues.apache.org/jira/browse/YARN-5590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev reassigned YARN-5590: --- Assignee: Manikandan R (was: Varun Vasudev) > Add support for increase and decrease of container resources with resource > profiles > --- > > Key: YARN-5590 > URL: https://issues.apache.org/jira/browse/YARN-5590 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Manikandan R >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5590) Add support for increase and decrease of container resources with resource profiles
[ https://issues.apache.org/jira/browse/YARN-5590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344526#comment-16344526 ] Varun Vasudev commented on YARN-5590: - [~maniraj...@gmail.com] - please feel free to take it up. This is to add support for increasing and decreasing container resources, to extend the functionality added by YARN-1197 to apply it to all resource types. > Add support for increase and decrease of container resources with resource > profiles > --- > > Key: YARN-5590 > URL: https://issues.apache.org/jira/browse/YARN-5590 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7430) Enable user re-mapping for Docker containers by default
[ https://issues.apache.org/jira/browse/YARN-7430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16262784#comment-16262784 ] Varun Vasudev commented on YARN-7430: - Backported to branch-3.0.0. Thanks for the quick repsonse [~andrew.wang]! > Enable user re-mapping for Docker containers by default > --- > > Key: YARN-7430 > URL: https://issues.apache.org/jira/browse/YARN-7430 > Project: Hadoop YARN > Issue Type: Sub-task > Components: security, yarn >Affects Versions: 2.9.0, 3.0.0 >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Blocker > Fix For: 3.0.0, 3.1.0, 2.10.0, 2.9.1 > > Attachments: YARN-7430.001.patch, YARN-7430.png > > > In YARN-4266, the recommendation was to use -u [uid]:[gid] numeric values to > enforce user and group for the running user. In YARN-6623, this translated > to --user=test --group-add=group1. The code no longer enforce group > correctly for launched process. > In addition, the implementation in YARN-6623 requires the user and group > information to exist in container to translate username and group to uid/gid. > For users on LDAP, there is no good way to populate container with user and > group information. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7430) Enable user re-mapping for Docker containers by default
[ https://issues.apache.org/jira/browse/YARN-7430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-7430: Fix Version/s: (was: 3.0.1) 3.0.0 > Enable user re-mapping for Docker containers by default > --- > > Key: YARN-7430 > URL: https://issues.apache.org/jira/browse/YARN-7430 > Project: Hadoop YARN > Issue Type: Sub-task > Components: security, yarn >Affects Versions: 2.9.0, 3.0.0 >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Blocker > Fix For: 3.0.0, 3.1.0, 2.10.0, 2.9.1 > > Attachments: YARN-7430.001.patch, YARN-7430.png > > > In YARN-4266, the recommendation was to use -u [uid]:[gid] numeric values to > enforce user and group for the running user. In YARN-6623, this translated > to --user=test --group-add=group1. The code no longer enforce group > correctly for launched process. > In addition, the implementation in YARN-6623 requires the user and group > information to exist in container to translate username and group to uid/gid. > For users on LDAP, there is no good way to populate container with user and > group information. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7430) Enable user re-mapping for Docker containers by default
[ https://issues.apache.org/jira/browse/YARN-7430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16262435#comment-16262435 ] Varun Vasudev commented on YARN-7430: - [~andrew.wang] - given that you're cutting a new RC for 3.0 , do you mind if we backport this into 3.0? It's a reasonably important configuration change and fixes the default behaviour when Docker container are launched. Thanks! > Enable user re-mapping for Docker containers by default > --- > > Key: YARN-7430 > URL: https://issues.apache.org/jira/browse/YARN-7430 > Project: Hadoop YARN > Issue Type: Sub-task > Components: security, yarn >Affects Versions: 2.9.0, 3.0.0 >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Blocker > Fix For: 3.1.0, 2.10.0, 2.9.1, 3.0.1 > > Attachments: YARN-7430.001.patch, YARN-7430.png > > > In YARN-4266, the recommendation was to use -u [uid]:[gid] numeric values to > enforce user and group for the running user. In YARN-6623, this translated > to --user=test --group-add=group1. The code no longer enforce group > correctly for launched process. > In addition, the implementation in YARN-6623 requires the user and group > information to exist in container to translate username and group to uid/gid. > For users on LDAP, there is no good way to populate container with user and > group information. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7430) Enable user re-mapping for Docker containers by default
[ https://issues.apache.org/jira/browse/YARN-7430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-7430: Summary: Enable user re-mapping for Docker containers by default (was: User and Group mapping are incorrect in docker container) > Enable user re-mapping for Docker containers by default > --- > > Key: YARN-7430 > URL: https://issues.apache.org/jira/browse/YARN-7430 > Project: Hadoop YARN > Issue Type: Sub-task > Components: security, yarn >Affects Versions: 2.9.0, 3.0.0 >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Blocker > Attachments: YARN-7430.001.patch, YARN-7430.png > > > In YARN-4266, the recommendation was to use -u [uid]:[gid] numeric values to > enforce user and group for the running user. In YARN-6623, this translated > to --user=test --group-add=group1. The code no longer enforce group > correctly for launched process. > In addition, the implementation in YARN-6623 requires the user and group > information to exist in container to translate username and group to uid/gid. > For users on LDAP, there is no good way to populate container with user and > group information. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7430) User and Group mapping are incorrect in docker container
[ https://issues.apache.org/jira/browse/YARN-7430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16255005#comment-16255005 ] Varun Vasudev commented on YARN-7430: - Looks like we have consensus of making uid:gid the default behavior. +1 from my end. I'll commit this tomorrow unless someone objects. > User and Group mapping are incorrect in docker container > > > Key: YARN-7430 > URL: https://issues.apache.org/jira/browse/YARN-7430 > Project: Hadoop YARN > Issue Type: Sub-task > Components: security, yarn >Affects Versions: 2.9.0, 3.0.0 >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Blocker > Attachments: YARN-7430.001.patch, YARN-7430.png > > > In YARN-4266, the recommendation was to use -u [uid]:[gid] numeric values to > enforce user and group for the running user. In YARN-6623, this translated > to --user=test --group-add=group1. The code no longer enforce group > correctly for launched process. > In addition, the implementation in YARN-6623 requires the user and group > information to exist in container to translate username and group to uid/gid. > For users on LDAP, there is no good way to populate container with user and > group information. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7430) User and Group mapping are incorrect in docker container
[ https://issues.apache.org/jira/browse/YARN-7430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250879#comment-16250879 ] Varun Vasudev commented on YARN-7430: - I'm fine with it. [~ebadger] - does it sounds ok to you? > User and Group mapping are incorrect in docker container > > > Key: YARN-7430 > URL: https://issues.apache.org/jira/browse/YARN-7430 > Project: Hadoop YARN > Issue Type: Sub-task > Components: security, yarn >Affects Versions: 2.9.0, 3.0.0 >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Blocker > Attachments: YARN-7430.001.patch, YARN-7430.png > > > In YARN-4266, the recommendation was to use -u [uid]:[gid] numeric values to > enforce user and group for the running user. In YARN-6623, this translated > to --user=test --group-add=group1. The code no longer enforce group > correctly for launched process. > In addition, the implementation in YARN-6623 requires the user and group > information to exist in container to translate username and group to uid/gid. > For users on LDAP, there is no good way to populate container with user and > group information. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7361) Improve the docker container runtime documentation
[ https://issues.apache.org/jira/browse/YARN-7361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250865#comment-16250865 ] Varun Vasudev commented on YARN-7361: - +1 - I'll commit this tomorrow if no one objects. > Improve the docker container runtime documentation > -- > > Key: YARN-7361 > URL: https://issues.apache.org/jira/browse/YARN-7361 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Attachments: YARN-7361.001.patch > > > During review of YARN-7230, it was found that > yarn.nodemanager.runtime.linux.docker.capabilities is missing from the docker > containers documentation in most of the active branches. We can also improve > the warning that was introduced in YARN-6622. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7430) User and Group mapping are incorrect in docker container
[ https://issues.apache.org/jira/browse/YARN-7430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241947#comment-16241947 ] Varun Vasudev commented on YARN-7430: - [~eyang] - can you modify your path slightly - if the user is running non-privileged containers, let the default be the current default(don't enable user mapping). If the user is running privileged containers, let the default be to enable user mapping, with a note added to the documentation about the difference in defaults between privileged and non-privileged containers. That way we don't change the current behavior. What do you think? > User and Group mapping are incorrect in docker container > > > Key: YARN-7430 > URL: https://issues.apache.org/jira/browse/YARN-7430 > Project: Hadoop YARN > Issue Type: Bug > Components: security, yarn >Affects Versions: 2.9.0, 3.0.0 >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Blocker > Attachments: YARN-7430.001.patch > > > In YARN-4266, the recommendation was to use -u [uid]:[gid] numeric values to > enforce user and group for the running user. In YARN-6623, this translated > to --user=test --group-add=group1. The code no longer enforce group > correctly for launched process. > In addition, the implementation in YARN-6623 requires the user and group > information to exist in container to translate username and group to uid/gid. > For users on LDAP, there is no good way to populate container with user and > group information. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7430) User and Group mapping are incorrect in docker container
[ https://issues.apache.org/jira/browse/YARN-7430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240104#comment-16240104 ] Varun Vasudev commented on YARN-7430: - [~shaneku...@gmail.com], [~ebadger] - any objections to turning on user-remapping by default? > User and Group mapping are incorrect in docker container > > > Key: YARN-7430 > URL: https://issues.apache.org/jira/browse/YARN-7430 > Project: Hadoop YARN > Issue Type: Bug > Components: security, yarn >Affects Versions: 2.9.0, 3.0.0 >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Blocker > Attachments: YARN-7430.001.patch > > > In YARN-4266, the recommendation was to use -u [uid]:[gid] numeric values to > enforce user and group for the running user. In YARN-6623, this translated > to --user=test --group-add=group1. The code no longer enforce group > correctly for launched process. > In addition, the implementation in YARN-6623 requires the user and group > information to exist in container to translate username and group to uid/gid. > For users on LDAP, there is no good way to populate container with user and > group information. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-6623: Release Note: A change in configuration for launching Docker containers under YARN. Docker container capabilities, mounts, networks and allowing privileged container have to specified in the container-executor.cfg. By default, all of the above are turned off. This change will break existing setups launching Docker containers under YARN. Please refer to the Docker containers under YARN documentation for more information. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Fix For: 2.9.0, 3.0.0 > > Attachments: YARN-6623-branch-2.013.patch, > YARN-6623-branch-2.014.patch, YARN-6623-branch-2.015.patch, > YARN-6623.001.patch, YARN-6623.002.patch, YARN-6623.003.patch, > YARN-6623.004.patch, YARN-6623.005.patch, YARN-6623.006.patch, > YARN-6623.007.patch, YARN-6623.008.patch, YARN-6623.009.patch, > YARN-6623.010.patch, YARN-6623.011.patch, YARN-6623.012.patch, > YARN-6623.013.patch, cetest.stderr, cetest.stdout > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6930) Admins should be able to explicitly enable specific LinuxContainerRuntime in the NodeManager
[ https://issues.apache.org/jira/browse/YARN-6930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16225289#comment-16225289 ] Varun Vasudev commented on YARN-6930: - [~djp], this change turns off the Docker runtime by default. It has to be explicitly turned on in 2.9. In 2.8 it was turned on by default .Given that the Docker work is alpha, it may not be necessary, but I just wanted to make sure. > Admins should be able to explicitly enable specific LinuxContainerRuntime in > the NodeManager > > > Key: YARN-6930 > URL: https://issues.apache.org/jira/browse/YARN-6930 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Vinod Kumar Vavilapalli >Assignee: Shane Kumpf > Fix For: 2.9.0, 3.0.0-beta1, 2.8.2 > > Attachments: YARN-6930.001.patch, YARN-6930.002.patch, > YARN-6930.003.patch, YARN-6930.004.patch, YARN-6930.005.patch, > YARN-6930.006.patch, YARN-6930.branch-2.001.patch, > YARN-6930.branch-2.002.patch, YARN-6930.branch-2.8.001.patch, > YARN-6930.branch-2.8.002.patch, YARN-6930.branch-2.8.2.001.patch > > > Today, in the java land, all LinuxContainerRuntimes are always enabled when > using LinuxContainerExecutor and the user can simply invoke anything that > he/she wants - default, docker, java-sandbox. > We should have a way for admins to explicitly enable only specific runtimes > that he/she decides for the cluster. And by default, we should have > everything other than the default one disabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6930) Admins should be able to explicitly enable specific LinuxContainerRuntime in the NodeManager
[ https://issues.apache.org/jira/browse/YARN-6930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16225117#comment-16225117 ] Varun Vasudev commented on YARN-6930: - [~jlowe]/[~shaneku...@gmail.com] - should this be marked as an incompatible change? Existing Docker configs will have to change to run with 2.9.0. > Admins should be able to explicitly enable specific LinuxContainerRuntime in > the NodeManager > > > Key: YARN-6930 > URL: https://issues.apache.org/jira/browse/YARN-6930 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Vinod Kumar Vavilapalli >Assignee: Shane Kumpf > Fix For: 2.9.0, 3.0.0-beta1, 2.8.2 > > Attachments: YARN-6930.001.patch, YARN-6930.002.patch, > YARN-6930.003.patch, YARN-6930.004.patch, YARN-6930.005.patch, > YARN-6930.006.patch, YARN-6930.branch-2.001.patch, > YARN-6930.branch-2.002.patch, YARN-6930.branch-2.8.001.patch, > YARN-6930.branch-2.8.002.patch, YARN-6930.branch-2.8.2.001.patch > > > Today, in the java land, all LinuxContainerRuntimes are always enabled when > using LinuxContainerExecutor and the user can simply invoke anything that > he/she wants - default, docker, java-sandbox. > We should have a way for admins to explicitly enable only specific runtimes > that he/she decides for the cluster. And by default, we should have > everything other than the default one disabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-6623: Hadoop Flags: Incompatible change,Reviewed (was: Reviewed) > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Fix For: 2.9.0, 3.0.0 > > Attachments: YARN-6623-branch-2.013.patch, > YARN-6623-branch-2.014.patch, YARN-6623-branch-2.015.patch, > YARN-6623.001.patch, YARN-6623.002.patch, YARN-6623.003.patch, > YARN-6623.004.patch, YARN-6623.005.patch, YARN-6623.006.patch, > YARN-6623.007.patch, YARN-6623.008.patch, YARN-6623.009.patch, > YARN-6623.010.patch, YARN-6623.011.patch, YARN-6623.012.patch, > YARN-6623.013.patch, cetest.stderr, cetest.stdout > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7376) YARN top ACLs
[ https://issues.apache.org/jira/browse/YARN-7376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16218080#comment-16218080 ] Varun Vasudev commented on YARN-7376: - [~jhung] -- this is implemented on the client side. The user can just modify the config file and change the ACL. For this to be work, it has to be implemented on the server side. > YARN top ACLs > - > > Key: YARN-7376 > URL: https://issues.apache.org/jira/browse/YARN-7376 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Attachments: YARN-7376.001.patch, YARN-7376.002.patch > > > Currently YARN top can be invoked by everyone. But we want to avoid a > scenario where random users invoke YARN top, and potentially leave it > running. So we can implement ACLs to prevent this. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7353) Docker permitted volumes don't properly check for directories
[
https://issues.apache.org/jira/browse/YARN-7353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16216291#comment-16216291
]
Varun Vasudev commented on YARN-7353:
-
[~eyang] - can you please cherry-pick this patch to branch-3 and branch-2 as
well? Thanks!
> Docker permitted volumes don't properly check for directories
> -
>
> Key: YARN-7353
> URL: https://issues.apache.org/jira/browse/YARN-7353
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: YARN-7353.001.patch, YARN-7353.002.patch,
> YARN-7353.003.patch
>
>
> {noformat:title=docker-util.c:check_mount_permitted()}
> // directory check
> permitted_mount_len = strlen(permitted_mounts[i]);
> if (permitted_mount_len > 0
> && permitted_mounts[i][permitted_mount_len - 1] == '/') {
> if (strncmp(normalized_path, permitted_mounts[i], permitted_mount_len)
> == 0) {
> ret = 1;
> break;
> }
> }
> {noformat}
> This code will treat "/home/" as a directory, but not "/home"
> {noformat}
> [ FAILED ] 3 tests, listed below:
> [ FAILED ] TestDockerUtil.test_check_mount_permitted
> [ FAILED ] TestDockerUtil.test_normalize_mounts
> [ FAILED ] TestDockerUtil.test_add_rw_mounts
> {noformat}
> Additionally, YARN-6623 introduced new test failures in the C++
> container-executor test "cetest"
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7353) Docker permitted volumes don't properly check for directories
[
https://issues.apache.org/jira/browse/YARN-7353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210587#comment-16210587
]
Varun Vasudev commented on YARN-7353:
-
Thanks for the patch [~ebadger]. A couple of tests still fail on Centos 7 due
to /bin being symlinked to /usr/bin - test_normalize_mounts and
test_add_rw_mounts.
Here are the changes I had made for YARN-7344 -
{noformat}
TEST_F(TestDockerUtil, test_normalize_mounts) {
const int entries = 4;
-const char *permitted_mounts[] = {"/home", "/usr", "/bin/ls", NULL};
-const char *expected[] = {"/home/", "/usr/", "/bin/ls", NULL};
+const char *permitted_mounts[] = {"/home", "/usr", "/usr/bin/yes", NULL};
+const char *expected[] = {"/home/", "/usr/", "/usr/bin/yes", NULL};
char **ptr = static_cast(malloc(entries * sizeof(char *)));
for (int i = 0; i < entries; ++i) {
if (permitted_mounts[i] != NULL) {
@@ -659,22 +659,22 @@ namespace ContainerExecutor {
const int buff_len = 1024;
char buff[buff_len];
int ret = 0;
-std::string container_executor_cfg_contents = "[docker]\n
docker.allowed.rw-mounts=/usr,/var,/bin/ls,..\n "
-
"docker.allowed.ro-mounts=/bin/cat";
+std::string container_executor_cfg_contents = "[docker]\n
docker.allowed.rw-mounts=/opt,/var,/usr/bin/yes,..\n "
+
"docker.allowed.ro-mounts=/usr/bin/cut";
std::vector > file_cmd_vec;
file_cmd_vec.push_back(std::make_pair(
"[docker-command-execution]\n docker-command=run\n
rw-mounts=/var:/var", "-v '/var:/var' "));
file_cmd_vec.push_back(std::make_pair(
"[docker-command-execution]\n docker-command=run\n
rw-mounts=/var/:/var/", "-v '/var/:/var/' "));
file_cmd_vec.push_back(std::make_pair(
-"[docker-command-execution]\n docker-command=run\n
rw-mounts=/usr:/usr", "-v '/usr:/usr' "));
+"[docker-command-execution]\n docker-command=run\n
rw-mounts=/opt:/opt", "-v '/opt:/opt' "));
file_cmd_vec.push_back(std::make_pair(
-"[docker-command-execution]\n docker-command=run\n
rw-mounts=/usr/:/usr", "-v '/usr/:/usr' "));
+"[docker-command-execution]\n docker-command=run\n
rw-mounts=/opt/:/opt", "-v '/opt/:/opt' "));
file_cmd_vec.push_back(std::make_pair(
-"[docker-command-execution]\n docker-command=run\n
rw-mounts=/bin/ls:/bin/ls", "-v '/bin/ls:/bin/ls' "));
+"[docker-command-execution]\n docker-command=run\n
rw-mounts=/usr/bin/yes:/usr/bin/yes", "-v '/usr/bin/yes:/usr/bin/yes' "));
file_cmd_vec.push_back(std::make_pair(
-"[docker-command-execution]\n docker-command=run\n
rw-mounts=/usr/bin:/mydisk1,/var/log/:/mydisk2",
-"-v '/usr/bin:/mydisk1' -v '/var/log/:/mydisk2' "));
+"[docker-command-execution]\n docker-command=run\n
rw-mounts=/opt:/mydisk1,/var/log/:/mydisk2",
+"-v '/opt:/mydisk1' -v '/var/log/:/mydisk2' "));
file_cmd_vec.push_back(std::make_pair(
"[docker-command-execution]\n docker-command=run\n", ""));
write_container_executor_cfg(container_executor_cfg_contents);
@@ -708,7 +708,7 @@ namespace ContainerExecutor {
"[docker-command-execution]\n docker-command=run\n
rw-mounts=/home:/home",
static_cast(INVALID_DOCKER_RW_MOUNT)));
bad_file_cmds_vec.push_back(std::make_pair(
-"[docker-command-execution]\n docker-command=run\n
rw-mounts=/bin/cat:/bin/cat",
+"[docker-command-execution]\n docker-command=run\n
rw-mounts=/usr/bin/cut:/usr/bin/cut",
static_cast(INVALID_DOCKER_RW_MOUNT)));
bad_file_cmds_vec.push_back(std::make_pair(
"[docker-command-execution]\n docker-command=run\n
rw-mounts=/blah:/blah",
{noformat}
Can you incorporate them into your patch? Thanks!
> Docker permitted volumes don't properly check for directories
> -
>
> Key: YARN-7353
> URL: https://issues.apache.org/jira/browse/YARN-7353
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: YARN-7353.001.patch, YARN-7353.002.patch
>
>
> {noformat:title=docker-util.c:check_mount_permitted()}
> // directory check
> permitted_mount_len = strlen(permitted_mounts[i]);
> if (permitted_mount_len > 0
> && permitted_mounts[i][permitted_mount_len - 1] == '/') {
> if (strncmp(normalized_path, permitted_mounts[i], permitted_mount_len)
> == 0) {
>
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16207265#comment-16207265 ] Varun Vasudev commented on YARN-6623: - [~leftnoteasy] - can you take a look at the branch-2 patch please? Thanks! > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Fix For: 3.0.0 > > Attachments: YARN-6623-branch-2.013.patch, > YARN-6623-branch-2.014.patch, YARN-6623-branch-2.015.patch, > YARN-6623.001.patch, YARN-6623.002.patch, YARN-6623.003.patch, > YARN-6623.004.patch, YARN-6623.005.patch, YARN-6623.006.patch, > YARN-6623.007.patch, YARN-6623.008.patch, YARN-6623.009.patch, > YARN-6623.010.patch, YARN-6623.011.patch, YARN-6623.012.patch, > YARN-6623.013.patch, cetest.stderr, cetest.stdout > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16207264#comment-16207264
]
Varun Vasudev commented on YARN-6623:
-
[~eyang] - the tests did run as part of the trunk pre-commit build. The test
reports are lost now but it will appear under the section (root) in the test
report.
The tests seem to pass for me on the Mac and an Ubuntu VM. What's the
environment you're using? Maybe you should open a new ticket with details?
Here's the result I got -
Mac
{noformat}
Varuns-MacBook-Pro:hadoop-yarn-server-nodemanager varun$ mvn test -Dtest=cetest
-Pnative
[INFO] Scanning for projects...
[INFO]
[INFO]
[INFO] Building Apache Hadoop YARN NodeManager 3.1.0-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @
hadoop-yarn-server-nodemanager ---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- hadoop-maven-plugins:3.1.0-SNAPSHOT:protoc (compile-protoc) @
hadoop-yarn-server-nodemanager ---
[INFO] No changes detected in protoc files, skipping generation.
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @
hadoop-yarn-server-nodemanager ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
hadoop-yarn-server-nodemanager ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 4 resources
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @
hadoop-yarn-server-nodemanager ---
[INFO] Compiling 3 source files to
/Users/varun/Workspace/apache/committer-rw/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/classes
[INFO]
[INFO] --- hadoop-maven-plugins:3.1.0-SNAPSHOT:cmake-compile (cmake-compile) @
hadoop-yarn-server-nodemanager ---
[INFO] Running cmake
/Users/varun/Workspace/apache/committer-rw/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src
-DHADOOP_CONF_DIR=../etc/hadoop -DJVM_ARCH_DATA_MODEL=64 -G Unix Makefiles
[INFO] with extra environment variables {
CFLAGS = ''
}
[INFO] Running make -j 8 VERBOSE=1
[INFO] cmake compilation finished successfully in 205 millisecond(s).
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (make) @ hadoop-yarn-server-nodemanager
---
[INFO] Executing tasks
main:
[copy] Copying 4 files to
/Users/varun/Workspace/apache/committer-rw/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/test
[INFO] Executed tasks
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @
hadoop-yarn-server-nodemanager ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 7 resources
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
hadoop-yarn-server-nodemanager ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-surefire-plugin:2.17:test (default-test) @
hadoop-yarn-server-nodemanager ---
[INFO]
[INFO] --- hadoop-maven-plugins:3.1.0-SNAPSHOT:cmake-test
(test-container-executor) @ hadoop-yarn-server-nodemanager ---
[INFO]
[INFO] --- hadoop-maven-plugins:3.1.0-SNAPSHOT:cmake-test (cetest) @
hadoop-yarn-server-nodemanager ---
[INFO] ---
[INFO] C M A K E B U I L D E RT E S T
[INFO] ---
[INFO] cetest: running
/Users/varun/Workspace/apache/committer-rw/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/test/cetest
--gtest_filter=-Perf.
--gtest_output=xml:/Users/varun/Workspace/apache/committer-rw/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/surefire-reports/TEST-cetest.xml
[INFO] with extra environment variables {}
[INFO] STATUS: SUCCESS after 87 millisecond(s).
[INFO] ---
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 4.263 s
[INFO] Finished at: 2017-10-17T15:43:32+05:30
[INFO] Final Memory: 35M/318M
[INFO]
{noformat}
Ubuntu 16
{noformat}
varun@nm-u16:~/Workspace/apache/dev/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager$
mvn test -Dtest=cetest -Pnative
[INFO] Scanning for projects...
[INFO]
[INFO]
[INFO] Building Apache Hadoop YARN NodeManager 3.1.0-SNAPSHOT
[INFO]
[jira] [Updated] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-6623: Attachment: YARN-6623-branch-2.015.patch New version(branch-2.015.patch) with compile fix. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Fix For: 3.0.0 > > Attachments: YARN-6623-branch-2.013.patch, > YARN-6623-branch-2.014.patch, YARN-6623-branch-2.015.patch, > YARN-6623.001.patch, YARN-6623.002.patch, YARN-6623.003.patch, > YARN-6623.004.patch, YARN-6623.005.patch, YARN-6623.006.patch, > YARN-6623.007.patch, YARN-6623.008.patch, YARN-6623.009.patch, > YARN-6623.010.patch, YARN-6623.011.patch, YARN-6623.012.patch, > YARN-6623.013.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-6623: Attachment: YARN-6623-branch-2.014.patch Uploaded a new version of the branch-2 patch to fix the compile error. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Fix For: 3.0.0 > > Attachments: YARN-6623-branch-2.013.patch, > YARN-6623-branch-2.014.patch, YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch, YARN-6623.011.patch, > YARN-6623.012.patch, YARN-6623.013.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-6623: Attachment: YARN-6623-branch-2.013.patch Attached a patch for branch-2. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Fix For: 3.0.0 > > Attachments: YARN-6623-branch-2.013.patch, YARN-6623.001.patch, > YARN-6623.002.patch, YARN-6623.003.patch, YARN-6623.004.patch, > YARN-6623.005.patch, YARN-6623.006.patch, YARN-6623.007.patch, > YARN-6623.008.patch, YARN-6623.009.patch, YARN-6623.010.patch, > YARN-6623.011.patch, YARN-6623.012.patch, YARN-6623.013.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5589) Update CapacitySchedulerConfiguration minimum and maximum calculations to consider all resource types
[ https://issues.apache.org/jira/browse/YARN-5589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16205236#comment-16205236 ] Varun Vasudev commented on YARN-5589: - [~lovekesh.bansal] - please feel free to take it over. > Update CapacitySchedulerConfiguration minimum and maximum calculations to > consider all resource types > - > > Key: YARN-5589 > URL: https://issues.apache.org/jira/browse/YARN-5589 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-5589) Update CapacitySchedulerConfiguration minimum and maximum calculations to consider all resource types
[ https://issues.apache.org/jira/browse/YARN-5589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev reassigned YARN-5589: --- Assignee: (was: Varun Vasudev) > Update CapacitySchedulerConfiguration minimum and maximum calculations to > consider all resource types > - > > Key: YARN-5589 > URL: https://issues.apache.org/jira/browse/YARN-5589 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7321) Backport container-executor changes from YARN-6852 to branch-2
[ https://issues.apache.org/jira/browse/YARN-7321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203108#comment-16203108 ] Varun Vasudev commented on YARN-7321: - Thank for the review [~leftnoteasy]! I've verified that the changes work correctly on branch-2. I ran tests with the regular process container and Docker containers and there are no regressions. > Backport container-executor changes from YARN-6852 to branch-2 > -- > > Key: YARN-7321 > URL: https://issues.apache.org/jira/browse/YARN-7321 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: YARN-7321-branch-2.001.patch > > > YARN-6852 added support for GPUs to container-executor. It also re-factored > the container-executor code to add support for modules. The non-GPU changes > need to be backported to branch-2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4943) Add support to collect actual resource usage from cgroups
[ https://issues.apache.org/jira/browse/YARN-4943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203066#comment-16203066 ] Varun Vasudev commented on YARN-4943: - [~miklos.szeg...@cloudera.com] - please feel free to take it over. Thanks! > Add support to collect actual resource usage from cgroups > - > > Key: YARN-4943 > URL: https://issues.apache.org/jira/browse/YARN-4943 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Varun Vasudev >Assignee: Varun Vasudev > > We should add support to collect actual resource usage from Cgroups(if > they're enabled) - it's more accurate and it can give you more detailed > information. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-4943) Add support to collect actual resource usage from cgroups
[ https://issues.apache.org/jira/browse/YARN-4943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev reassigned YARN-4943: --- Assignee: (was: Varun Vasudev) > Add support to collect actual resource usage from cgroups > - > > Key: YARN-4943 > URL: https://issues.apache.org/jira/browse/YARN-4943 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Varun Vasudev > > We should add support to collect actual resource usage from Cgroups(if > they're enabled) - it's more accurate and it can give you more detailed > information. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7321) Backport container-executor changes from YARN-6852 to branch-2
[ https://issues.apache.org/jira/browse/YARN-7321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202339#comment-16202339 ] Varun Vasudev commented on YARN-7321: - [~wangda] - can you please take a look? Thanks! > Backport container-executor changes from YARN-6852 to branch-2 > -- > > Key: YARN-7321 > URL: https://issues.apache.org/jira/browse/YARN-7321 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: YARN-7321-branch-2.001.patch > > > YARN-6852 added support for GPUs to container-executor. It also re-factored > the container-executor code to add support for modules. The non-GPU changes > need to be backported to branch-2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7321) Backport container-executor changes from YARN-6852 to branch-2
[ https://issues.apache.org/jira/browse/YARN-7321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-7321: Attachment: YARN-7321-branch-2.001.patch Attached a branch-2 version of the patch. It just drops all the gpu and cgroups module files from YARN-6852, > Backport container-executor changes from YARN-6852 to branch-2 > -- > > Key: YARN-7321 > URL: https://issues.apache.org/jira/browse/YARN-7321 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: YARN-7321-branch-2.001.patch > > > YARN-6852 added support for GPUs to container-executor. It also re-factored > the container-executor code to add support for modules. The non-GPU changes > need to be backported to branch-2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7321) Backport container-executor changes from YARN-6852 to branch-2
Varun Vasudev created YARN-7321: --- Summary: Backport container-executor changes from YARN-6852 to branch-2 Key: YARN-7321 URL: https://issues.apache.org/jira/browse/YARN-7321 Project: Hadoop YARN Issue Type: Bug Reporter: Varun Vasudev Assignee: Varun Vasudev YARN-6852 added support for GPUs to container-executor. It also re-factored the container-executor code to add support for modules. The non-GPU changes need to be backported to branch-2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-6852) [YARN-6223] Native code changes to support isolate GPU devices by using CGroups
[ https://issues.apache.org/jira/browse/YARN-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev resolved YARN-6852. - Resolution: Fixed > [YARN-6223] Native code changes to support isolate GPU devices by using > CGroups > --- > > Key: YARN-6852 > URL: https://issues.apache.org/jira/browse/YARN-6852 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Fix For: 3.0.0-beta1 > > Attachments: YARN-6852.001.patch, YARN-6852.002.patch, > YARN-6852.003.patch, YARN-6852.004.patch, YARN-6852.005.patch, > YARN-6852.006.patch, YARN-6852.007.patch, YARN-6852.008.patch, > YARN-6852.009.patch > > > This JIRA plan to add support of: > 1) Isolation in CGroups. (native side). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6852) [YARN-6223] Native code changes to support isolate GPU devices by using CGroups
[ https://issues.apache.org/jira/browse/YARN-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202313#comment-16202313 ] Varun Vasudev commented on YARN-6852: - Sounds good, I'll create a new ticket for that then and close this. > [YARN-6223] Native code changes to support isolate GPU devices by using > CGroups > --- > > Key: YARN-6852 > URL: https://issues.apache.org/jira/browse/YARN-6852 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Fix For: 3.0.0-beta1 > > Attachments: YARN-6852.001.patch, YARN-6852.002.patch, > YARN-6852.003.patch, YARN-6852.004.patch, YARN-6852.005.patch, > YARN-6852.006.patch, YARN-6852.007.patch, YARN-6852.008.patch, > YARN-6852.009.patch > > > This JIRA plan to add support of: > 1) Isolation in CGroups. (native side). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Reopened] (YARN-6852) [YARN-6223] Native code changes to support isolate GPU devices by using CGroups
[ https://issues.apache.org/jira/browse/YARN-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev reopened YARN-6852: - [~leftnoteasy] - Any objections to backporting this to branch-2? It has changes to container-executor that I would like to pull to branch-2. Thanks! > [YARN-6223] Native code changes to support isolate GPU devices by using > CGroups > --- > > Key: YARN-6852 > URL: https://issues.apache.org/jira/browse/YARN-6852 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Fix For: 3.0.0-beta1 > > Attachments: YARN-6852.001.patch, YARN-6852.002.patch, > YARN-6852.003.patch, YARN-6852.004.patch, YARN-6852.005.patch, > YARN-6852.006.patch, YARN-6852.007.patch, YARN-6852.008.patch, > YARN-6852.009.patch > > > This JIRA plan to add support of: > 1) Isolation in CGroups. (native side). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6033) Add support for sections in container-executor configuration file
[ https://issues.apache.org/jira/browse/YARN-6033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199968#comment-16199968 ] Varun Vasudev commented on YARN-6033: - [~leftnoteasy] - do you mind committing this to branch-2? Thanks! > Add support for sections in container-executor configuration file > - > > Key: YARN-6033 > URL: https://issues.apache.org/jira/browse/YARN-6033 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 3.0.0-beta1 > > Attachments: YARN-6033-YARN-5673.001.patch, > YARN-6033-YARN-5673.002.patch, YARN-6033-branch-2.014.patch, > YARN-6033.003.patch, YARN-6033.004.patch, YARN-6033.005.patch, > YARN-6033.006.patch, YARN-6033.007.patch, YARN-6033.008.patch, > YARN-6033.009.patch, YARN-6033.010.patch, YARN-6033.011.patch, > YARN-6033.012.patch, YARN-6033.013.patch, YARN-6033.014.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7286) Add support for docker to have no capabilities
[
https://issues.apache.org/jira/browse/YARN-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199967#comment-16199967
]
Varun Vasudev commented on YARN-7286:
-
The none approach makes best sense to me. The
DEFAULT_NM_DOCKER_CONTAINER_CAPABILITIES was added because it's unreasonable to
expect every user to specify capabilities. We should just call it out in the
documentation that "none" is a special property for us. As to lower-case vs
upper-case, the upper case is just a recommendation, we should go with whatever
we think makes sense.
> Add support for docker to have no capabilities
> --
>
> Key: YARN-7286
> URL: https://issues.apache.org/jira/browse/YARN-7286
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: YARN-7286.001.patch, YARN-7286.002.patch,
> YARN-7286.003.patch
>
>
> Support for controlling capabilities was introduced in YARN-4258. However, it
> does not allow for the capabilities list to be NULL, since {{getStrings()}}
> will treat an empty value the same as it treats an unset property. So, a NULL
> list will actually give the default capabilities list.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6033) Add support for sections in container-executor configuration file
[ https://issues.apache.org/jira/browse/YARN-6033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-6033: Attachment: YARN-6033-branch-2.014.patch Patch for branch-2. It also takes the modifications for hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/cmakebuilder/TestMojo.java from HADOOP-12714 to make sure tests run correctly. > Add support for sections in container-executor configuration file > - > > Key: YARN-6033 > URL: https://issues.apache.org/jira/browse/YARN-6033 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 3.0.0-beta1 > > Attachments: YARN-6033-YARN-5673.001.patch, > YARN-6033-YARN-5673.002.patch, YARN-6033-branch-2.014.patch, > YARN-6033.003.patch, YARN-6033.004.patch, YARN-6033.005.patch, > YARN-6033.006.patch, YARN-6033.007.patch, YARN-6033.008.patch, > YARN-6033.009.patch, YARN-6033.010.patch, YARN-6033.011.patch, > YARN-6033.012.patch, YARN-6033.013.patch, YARN-6033.014.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Reopened] (YARN-6033) Add support for sections in container-executor configuration file
[ https://issues.apache.org/jira/browse/YARN-6033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev reopened YARN-6033: - Re-opening for branch-2 version of the patch. > Add support for sections in container-executor configuration file > - > > Key: YARN-6033 > URL: https://issues.apache.org/jira/browse/YARN-6033 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 3.0.0-beta1 > > Attachments: YARN-6033-YARN-5673.001.patch, > YARN-6033-YARN-5673.002.patch, YARN-6033.003.patch, YARN-6033.004.patch, > YARN-6033.005.patch, YARN-6033.006.patch, YARN-6033.007.patch, > YARN-6033.008.patch, YARN-6033.009.patch, YARN-6033.010.patch, > YARN-6033.011.patch, YARN-6033.012.patch, YARN-6033.013.patch, > YARN-6033.014.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16192875#comment-16192875
]
Varun Vasudev commented on YARN-6623:
-
{quote}
There's already a config in yarn-site.xml
"yarn.nodemanager.runtime.linux.docker.allowed-container-networks"
Will the one in yarn-site.xml be deprecated ? what's the story here?
{quote}
It can be deprecated or used for fast-failing containers. I'm open to either
option.
bq. Also, should the default value include host and bridge like the
yarn-defalut.xml did ? otherwise, the default setting will just not work, user
has to explicitly chage it
No, the idea is to have admins explicitly change it.
> Add support to turn off launching privileged containers in the
> container-executor
> -
>
> Key: YARN-6623
> URL: https://issues.apache.org/jira/browse/YARN-6623
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
>Priority: Blocker
> Fix For: 3.0.0
>
> Attachments: YARN-6623.001.patch, YARN-6623.002.patch,
> YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch,
> YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch,
> YARN-6623.009.patch, YARN-6623.010.patch, YARN-6623.011.patch,
> YARN-6623.012.patch, YARN-6623.013.patch
>
>
> Currently, launching privileged containers is controlled by the NM. We should
> add a flag to the container-executor.cfg allowing admins to disable launching
> privileged containers at the container-executor level.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16185866#comment-16185866 ] Varun Vasudev commented on YARN-6623: - Thanks for the reviews [~ebadger], [~miklos.szeg...@cloudera.com], [~shaneku...@gmail.com], [~sunil.gov...@gmail.com] and [~leftnoteasy]! I'll get a patch together for branch-2. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Fix For: 3.0.0-beta1 > > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch, YARN-6623.011.patch, > YARN-6623.012.patch, YARN-6623.013.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183909#comment-16183909 ] Varun Vasudev edited comment on YARN-6623 at 9/28/17 9:33 AM: -- [~andrew.wang] - the one thing you should be aware of is that YARN-6623 will break backwards compatibility from earlier releases because it disables most Docker features by default. So folks upgrading from beta-1 to beta-2 will find their Docker containers will no longer work. If you're fine with that, please go ahead with beta-1. was (Author: vvasudev): [~andrew.wang] - the one thing you should be aware of is that YARN-6623 will break backwards compatibility because from earlier releases because it disables most Docker features by default. So folks upgrading from beta-1 to beta-2 will find their Docker containers will no longer work. If you're fine with that, please go ahead with beta-1. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch, YARN-6623.011.patch, > YARN-6623.012.patch, YARN-6623.013.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183909#comment-16183909 ] Varun Vasudev commented on YARN-6623: - [~andrew.wang] - the one thing you should be aware of is that YARN-6623 will break backwards compatibility because from earlier releases because it disables most Docker features by default. So folks upgrading from beta-1 to beta-2 will find their Docker containers will no longer work. If you're fine with that, please go ahead with beta-1. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch, YARN-6623.011.patch, > YARN-6623.012.patch, YARN-6623.013.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-6623: Attachment: YARN-6623.013.patch Uploaded the wrong patch file by mistake. YARN-6623.013.patch is correct patch. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch, YARN-6623.011.patch, > YARN-6623.012.patch, YARN-6623.013.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-6623: Attachment: (was: YARN-6623.001.patch) > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch, YARN-6623.011.patch, > YARN-6623.012.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Vasudev updated YARN-6623:
Attachment: YARN-6623.001.patch
Thanks for the review [~ebadger].
{noformat}
Hey Varun Vasudev, thanks for the updated patch! The changes related to
YARN-4266 look good. Just had one comment on the code.
1850 +static int set_network(const struct configuration *command_config,
1851 + const struct configuration *conf, char *out,
1852 + const size_t outlen) {
1853 +
1854 + int ret = 0;
1855 + ret = add_param_to_command_if_allowed(command_config, conf, "net",
1856 +"docker.allowed.networks",
"--net=",
1857 +0, 0, out, outlen);
1858 + if (ret != 0) {
1859 +fprintf(ERRORFILE, "Could not find requested network in allowed
networks\n");
1860 +ret = INVALID_DOCKER_NETWORK;
1861 + }
1862 + if (ret != 0) {
1863 +memset(out, 0, outlen);
1864 + }
1865 + return ret;
1866 +}
Something I noticed in passing. We should combine the if(ret != 0) statements
here. This same thing also occurs in set_capabilities() and set_devices().
{noformat}
Fixed.
> Add support to turn off launching privileged containers in the
> container-executor
> -
>
> Key: YARN-6623
> URL: https://issues.apache.org/jira/browse/YARN-6623
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
>Priority: Blocker
> Attachments: YARN-6623.001.patch, YARN-6623.001.patch,
> YARN-6623.002.patch, YARN-6623.003.patch, YARN-6623.004.patch,
> YARN-6623.005.patch, YARN-6623.006.patch, YARN-6623.007.patch,
> YARN-6623.008.patch, YARN-6623.009.patch, YARN-6623.010.patch,
> YARN-6623.011.patch, YARN-6623.012.patch
>
>
> Currently, launching privileged containers is controlled by the NM. We should
> add a flag to the container-executor.cfg allowing admins to disable launching
> privileged containers at the container-executor level.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-6623: Attachment: YARN-6623.012.patch Uploaded 012.patch to fix cc and unit test issues. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch, YARN-6623.011.patch, > YARN-6623.012.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-6623: Attachment: YARN-6623.011.patch Uploaded a new patch (.011). It adds support for group-add and adds documentation for the container-executor.cfg settings. [~ebadger] - can you please take a look at the group-add changes and confirm they're ok? > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch, YARN-6623.011.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6504) Add support for resource profiles in MapReduce
[ https://issues.apache.org/jira/browse/YARN-6504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178183#comment-16178183 ] Varun Vasudev commented on YARN-6504: - I had a bunch of conversations about whether we should support override capabilities. My reason for adding it in the end is that without overrides, every existing job has to be re-written and re-tested, etc to match the profiles the admin creates. However, I'm in heavy favor of allowing overrides for memory and vcores only and not letting users override other resource types. > Add support for resource profiles in MapReduce > -- > > Key: YARN-6504 > URL: https://issues.apache.org/jira/browse/YARN-6504 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: YARN-6504-YARN-3926.001.patch, > YARN-6504.YARN-3926.002.patch, YARN-6504.YARN-3926.003.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16177577#comment-16177577 ] Varun Vasudev commented on YARN-6623: - [~andrew.wang] - YARN-4266 adds a new field for the Docker invocation so I need to update the patch to add it in. [~djp] - the discussion made sense to me. Please go ahead and file that JIRA. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16177578#comment-16177578 ] Varun Vasudev commented on YARN-6623: - [~andrew.wang] - YARN-4266 adds a new field for the Docker invocation so I need to update the patch to add it in. [~djp] - the discussion made sense to me. Please go ahead and file that JIRA. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176346#comment-16176346 ] Varun Vasudev edited comment on YARN-6623 at 9/22/17 12:41 PM: --- [~leftnoteasy] - please don't commit the patch. I need to incorporate YARN-4266 into this. I should have a patch for you over the weekend. Thanks! was (Author: vvasudev): [~leftnoteasy] - please don't commit the patch tomorrow. I need to incorporate YARN-4266 into this. I should have a patch for you over the weekend. Thanks! > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176346#comment-16176346 ] Varun Vasudev commented on YARN-6623: - [~leftnoteasy] - please don't commit the patch tomorrow. I need to incorporate YARN-4266 into this. I should have a patch for you over the weekend. Thanks! > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175201#comment-16175201
]
Varun Vasudev commented on YARN-6623:
-
{noformat}
How would you detect the condition where the buffer doesn't have enough
size?
You copy at most bufflen-strlen(buff) characters including \0. As I said only
one strlen is enough in this case.
{noformat}
I think I understand what you're saying. The current implementation checks if
the buffer has enough space to hold the final string and will return an error
if there isn't enough space. Without the additional strlen, how would I check
that the buffer can fit the additional string?
{noformat}
381 quote_and_append_arg(_buffer, _buffer_size, " ", image_name);
That space might need to be added to the quote_and_append_arg function for
safety reasons.
I didn't get this. Can you please explain?
When we add a new arg then the space should be added by default in
quote_and_append_arg every time.
{noformat}
Ah I see what you're saying. Thanks for the explanation. That line of code got
removed in the latest patch to address the feedback from
[~shaneku...@gmail.com] in his review.
{noformat}
Is there any benefit to strcpy + strcat?
There is no need to do an strlen. Was the intention maybe to bound by the size
of tmp_buffer_2?
{noformat}
Yep. The thinking was to limit it to the size of tmp_buffer_2.
> Add support to turn off launching privileged containers in the
> container-executor
> -
>
> Key: YARN-6623
> URL: https://issues.apache.org/jira/browse/YARN-6623
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
>Priority: Blocker
> Attachments: YARN-6623.001.patch, YARN-6623.002.patch,
> YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch,
> YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch,
> YARN-6623.009.patch, YARN-6623.010.patch
>
>
> Currently, launching privileged containers is controlled by the NM. We should
> add a flag to the container-executor.cfg allowing admins to disable launching
> privileged containers at the container-executor level.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175181#comment-16175181 ] Varun Vasudev commented on YARN-6623: - [~djp] - makes sense to push it into 2.8.2; [~andrew.wang] - thanks for updating the target versions. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Vasudev updated YARN-6623:
Attachment: YARN-6623.010.patch
Thanks for the reviews [~miklos.szeg...@cloudera.com] and
[~shaneku...@gmail.com]!
I've uploaded a new patch to address your comments.
{noformat}
84 printWriter.println(" " + entry.getKey() + "=" + StringUtils
85 .join(",", entry.getValue()));
writeCommandToTempFile: entry.getKey() can still contain an = in the latest
patch, which is an issue. It probably needs to be filtered in
addCommandArguments.
{noformat}
Fixed.
{noformat}
701char *get_config_path(const char *argv0) {
702 char *executable_file = get_executable((char *) argv0);
703 if (!executable_file) {
704fprintf(ERRORFILE, "realpath of executable: %s\n",
705errno != 0 ? strerror(errno) : "unknown");
706return NULL;
707 }
It is probably a good idea to check for executable_file[0] != 0 as well
{noformat}
Fixed.
{noformat}
1150 size_t command_size = MIN(sysconf(_SC_ARG_MAX), 128*1024);
1151 char *buffer = alloc_and_clear_memory(command_size, sizeof(char));
1152 ret = get_docker_command(command_file, , buffer,
EXECUTOR_PATH_MAX);
The code passes in a different size than the actual size of the buffer.
{noformat}
Good catch! Fixed.
{noformat}
157inline void* alloc_and_clear_memory(size_t num, size_t size) {
158 void *ret = calloc(num, size);
159 if (ret == NULL) {
160exit(OUT_OF_MEMORY);
161 }
162 return ret;
163}
It might be a good idea to print an error message here.
{noformat}
Fixed.
{noformat}
42static int add_to_buffer(char *buff, const size_t bufflen, const char
*string) {
Why do not you use strncat inside? It would spare one of the strlen’s.
{noformat}
How would you detect the condition where the buffer doesn't have enough size?
{noformat}
105if(prefix != 0) {
106 tmp_ptr = strchr(values[i], prefix);
107 if (tmp_ptr == NULL) {
...
This feels a little bit less readable. I would suggest having a len instead of
tmp_ptr defaulted to strlen(tmp_ptr); Also, am I right that we are checking
only if the left device is allowed?
{noformat}
I didn't quite understand this. What would the len do? Your understanding is
correct, we're checking if the left device is allowed.
{noformat}
150 if (ret != 0) {
151memset(out, 0, outlen);
152 }
out[0]=0 should be sufficient, if outlen > 0 and ret != 0
{noformat}
I prefer to memset the entire buffer.
{noformat}
162 if (0 == strncmp(container_name, CONTAINER_NAME_PREFIX,
strlen(CONTAINER_NAME_PREFIX))) {
There is no need of an strlen here, sizeof is sufficient and calculates compile
time
{noformat}
I tried to change this to sizeof but I got a compiler warning complaining about
it.
{noformat}
283 ret = add_docker_config_param(_config, out, outlen);
284 if (ret != 0) {
285 return BUFFER_TOO_SMALL;
Container name is not freed.
{noformat}
Fixed.
{noformat}
330 if (ret != 0) {
331 return BUFFER_TOO_SMALL;
332 }
Image name is not freed.
{noformat}
Fixed.
{noformat}
381 quote_and_append_arg(_buffer, _buffer_size, " ", image_name);
That space might need to be added to the quote_and_append_arg function for
safety reasons.
{noformat}
I didn't get this. Can you please explain?
{noformat}
564 * 2. If the path is a directory, add a '/' at the end( if not present)
There is a small typo here.
{noformat}
Fixed.
{noformat}
585 if (len <= 0) {
586 return NULL;
587 }
There is a missing free here
{noformat}
Fixed.
{noformat}
731 strncpy(tmp_buffer_2, values[i], strlen(values[i]));
732 strncpy(tmp_buffer_2 + strlen(values[i]), ro_suffix, strlen(ro_suffix));
Why do you use strncpy here? Why not strcpy and strcat?
{noformat}
Is there any benefit to strcpy + strcat?
{noformat}
Can we print out the mount that was requested but failed for the read-only and
read write checks? I found it a bit difficult to determine which mount was the
cause of the failure. Doing the same for the devices option would be helpful as
well. Something like the following:
if (permitted_rw == 0) {
fprintf(ERRORFILE, "Requested rw mount not found in the rw whitelist
'%s', realpath=%s\n", values[i], mount_src);
ret = INVALID_DOCKER_RW_MOUNT;
goto free_and_exit;
-snip-
if (ro != 0 && permitted_ro == 0 && permitted_rw == 0) {
fprintf(ERRORFILE, "Requested ro mount not found in the ro whitelist
'%s', realpath=%s\n", values[i], mount_src);
ret = INVALID_DOCKER_RO_MOUNT;
goto free_and_exit;
}
{noformat}
Fixed.
{noformat}
Currently only docker run works with this patch. I know you are aware of this.
There are a few regressions in this patch from what was fixed in YARN-6726. It
would be great if we could try to address them here. I'll list the
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170280#comment-16170280 ] Varun Vasudev commented on YARN-6623: - Thanks for the feedback [~miklos.szeg...@cloudera.com], [~shaneku...@gmail.com]. I should have a patch addressing you reviews soon. [~miklos.szeg...@cloudera.com] bq. I have some comments but before that, does the jira target 3.0-beta1? I would like to get it into beta1. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-6623: Attachment: YARN-6623.009.patch bq. git apply is failing for me on DockerClient.java. It looks like it doesn't do fuzzing like patch does. Fixed. bq. Additionally, the native code doesn't compile for me on my rhel 6.8 VM. It looks like it's complaining about the C99 syntax that Miklos Szegedi mentioned here. Weird. I've changed the initialization, should be fixed in the latest patch. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16157018#comment-16157018 ] Varun Vasudev edited comment on YARN-6623 at 9/7/17 2:37 PM: - bq. git apply is failing for me on DockerClient.java. It looks like it doesn't do fuzzing like patch does. Fixed. bq. Additionally, the native code doesn't compile for me on my rhel 6.8 VM. It looks like it's complaining about the C99 syntax that Miklos Szegedi mentioned here. Weird. I've changed the initialization, should be fixed in the latest patch. was (Author: vvasudev): bq. git apply is failing for me on DockerClient.java. It looks like it doesn't do fuzzing like patch does. Fixed. bq. Additionally, the native code doesn't compile for me on my rhel 6.8 VM. It looks like it's complaining about the C99 syntax that Miklos Szegedi mentioned here. Weird. I've changed the initialization, should be fixed in the latest patch. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Vasudev updated YARN-6623:
Attachment: YARN-6623.008.patch
{noformat}
int is_docker_support_enabled() {
-return is_feature_enabled(DOCKER_SUPPORT_ENABLED_KEY,
-DEFAULT_DOCKER_SUPPORT_ENABLED, _cfg);
+ return is_feature_enabled(DOCKER_SUPPORT_ENABLED_KEY,
+DEFAULT_DOCKER_SUPPORT_ENABLED, _cfg)
+ || docker_module_enabled();
The indentation here should be fixed.
{noformat}
Fixed.
{noformat}
+# The configs below deal with settings for Docker
+#[docker]
+# module.enabled=## enable/disable the module. set to "true" to enable,
disabled by default
+# docker.binary=/usr/bin/docker
+# allowed.capabilities=## comma seperated capabilities that can be granted,
e.g
CHOWN,DAC_OVERRIDE,FSETID,FOWNER,MKNOD,NET_RAW,SETGID,SETUID,SETFCAP,SETPCAP,NET_BIND_SERVICE,SYS_CHROOT,KILL,AUDIT_WRITE
+# allowed.devices=## comma seperated list of devices that can be mounted into
a container
+# allowed.networks=## comma seperated networks that can be used. e.g
bridge,host,none
+# allowed.ro-mounts=## comma seperated volumes that can be mounted as
read-only
+# allowed.rw-mounts=## comma seperate volumes that can be mounted as
read-write, add the yarn local and log dirs to this list to run Hadoop jobs
For the Docker configs, I think it'd be more clear if they were prefixed with
docker.
{noformat}
Fixed.
{noformat}
+
+
+
+
+
+
Since we're going to remove the hard-coded string in YARN-6968, should we leave
the findbugs warning and take care of it there?
{noformat}
Makes sense, removed.
{noformat}
+ protected final void addCommandArguments(String key, String value) {
+if (commandArguments.containsKey(key)) {
+ commandArguments.get(key).add(value);
+ return;
+}
No need to call contains() and then call get(). We can just call get and then
add the value if the return value isn't null.
{noformat}
Fixed.
{noformat}
+ @Override
+ public String toString() {
+StringBuffer ret = new StringBuffer("");
+ret.append(this.command);
Any reason not to create the string with the command in the constructor?
Something like
StringBuffer ret = new StringBuffer(this.command);
{noformat}
Fixed.
> Add support to turn off launching privileged containers in the
> container-executor
> -
>
> Key: YARN-6623
> URL: https://issues.apache.org/jira/browse/YARN-6623
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
>Priority: Blocker
> Attachments: YARN-6623.001.patch, YARN-6623.002.patch,
> YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch,
> YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch
>
>
> Currently, launching privileged containers is controlled by the NM. We should
> add a flag to the container-executor.cfg allowing admins to disable launching
> privileged containers at the container-executor level.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6504) Add support for resource profiles in MapReduce
[ https://issues.apache.org/jira/browse/YARN-6504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147550#comment-16147550 ] Varun Vasudev commented on YARN-6504: - bq. Any progress on this? Sorry for the late reply [~templedf]. I haven't gotten a chance to wrap this up. I'd like to get YARN-6623 out of the way first. Please feel free to re-assign if someone would like to work on it. > Add support for resource profiles in MapReduce > -- > > Key: YARN-6504 > URL: https://issues.apache.org/jira/browse/YARN-6504 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: YARN-6504-YARN-3926.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Vasudev updated YARN-6623:
Attachment: YARN-6623.007.patch
Thanks for the additional feedback [~ebadger] and [~leftnoteasy]!
{quote}
As an additional comment on my second pass, I don’t think need to call
normalize_mounts() up front since values might not have anything in it. Also,
we might have only RO or only RW mounts, so we don’t need to necessarily
compute permitted_rw or permitted_ro. We only need to if we’re actually going
to mount RW or RO respectively.
{quote}
Makes sense. Fixed.
{quote}
Just one quick comment: for feature enable/disable. I suggest to put under
section.
For GPU feature (YARN-6852), I made config like:
\[gpu]
# Enable / disable the module
module.enabled=true/false
If you're OK with the style, I suggest to make it consistent: You can use
module-configs.h to call module_enabled and get the config.
{quote}
Fixed.
bq. And what is feature tc? Could you make the feature name more descriptive?
tc is the traffic control piece for network throttling. I'll fix the
documentation in a follow up patch.
> Add support to turn off launching privileged containers in the
> container-executor
> -
>
> Key: YARN-6623
> URL: https://issues.apache.org/jira/browse/YARN-6623
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
>Priority: Blocker
> Attachments: YARN-6623.001.patch, YARN-6623.002.patch,
> YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch,
> YARN-6623.006.patch, YARN-6623.007.patch
>
>
> Currently, launching privileged containers is controlled by the NM. We should
> add a flag to the container-executor.cfg allowing admins to disable launching
> privileged containers at the container-executor level.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Vasudev updated YARN-6623:
Attachment: YARN-6623.006.patch
Thank you for the review [~ebadger]!
bq. Should we fix the no newline at the end of file warnings? The apply tool
complains about them.
Fixed.
{noformat}
DockerCommand:getCommandArguments()
public Map getCommandArguments() {
return Collections.unmodifiableMap(commandArguments);
}
This will return the command as well as the arguments. Unless we are
considering the /usr/docker to be the actual command and inspect to be one of
the arguments. If that’s what we’re expecting to happen here, then the name is
a little bit misleading. This might be more of a problem with how
commandArguments is named than how this function is named.
{noformat}
Renamed function to getCommandWithArguments. Is that ok?
{noformat}
container-executor.c:construct_docker_command()
+char *construct_docker_command(const char *command_file) {
+ int ret = 0;
+ char *buffer = (char *) malloc(EXECUTOR_PATH_MAX * sizeof(char));
This should use _SC_ARG_MAX as we did in YARN-6988
size_t command_size = MIN(sysconf(_SC_ARG_MAX), 128*1024);
Also, why not use calloc() instead of malloc() and then memset()?
container-executor.c:run_docker()
{noformat}
Fixed; changed to use the alloc_and_clear function.
{noformat}
+ docker_command = construct_docker_command(command_file);
+ docker_binary = get_docker_binary();
I don’t see these getting freed. Am I missing this invocation somewhere?
container-executor.c:run_docker()
{noformat}
We call the execvp function, the running program will get replaced by the
docker invocation.
{noformat}
+ memset(docker_command_with_binary, 0, EXECUTOR_PATH_MAX);
Is this necessary? We allocate the memory with calloc() which already clears
all of the memory upon allocation.
{noformat}
Yep. Fixed.
{noformat}
{
container-executor.h
// get the executable's filename
char* get_executable(char *argv0);
Do we need this declaration (in container-executor.h) since we have moved that
declaration into get_executable.h? We should also add get_executable.h in the
appropriate places. Looks like main.c and test-container-executor.c both call
get_executable.
{noformat}
You're correct; fixed
{noformat}
main.c:assert_valid_setup()
-fprintf(ERRORFILE,"realpath of executable: %s\n",
- errno != 0 ? strerror(errno) : "unknown");
-flush_and_close_log_files();
-exit(-1);
+fprintf(ERRORFILE, "realpath of executable: %s\n",
+errno != 0 ? strerror(errno) : "unknown");
+exit(INVALID_CONFIG_FILE);
Why don’t we want to flush the log files anymore?
{noformat}
Fixed.
{noformat}
util.c:alloc_and_clear_memory()
+void* alloc_and_clear_memory(size_t num, size_t size) {
+ void *ret = calloc(num, size);
+ if (ret == NULL) {
+exit(OUT_OF_MEMORY);
+ }
+ return ret;
+}
Should we inline this? Also, if we’re going to use this function, then all
calloc calls should be replaced with this (like the ones I mentioned above)
util.h
{noformat}
Fixed(made function inline and replaced calloc invocations with alloc_and_clear)
{noformat}
// DOCKER_CONTAINER_NAME_INVALID = 41,
Should add (NOT USED) to follow convention
docker-util.c:read_and_verify_command_file()
{noformat}
Fixed.
{noformat}
if (command == NULL || (strcmp(command, docker_command) != 0)) {
ret = INCORRECT_COMMAND;
}
Is command guaranteed to be null-terminated? It comes from the configuration
file, which is a Java creation and I don’t think Java null-terminates. This
comment is probably relevant for quite a few other places that are doing string
operations. We need to be very safe about this or else we risk possibly
overrunning into random regions of the heap. A safe alternative would be to use
the “n” version of all the string operations. This patch uses a mixed bag of
the regular versions and their accompanying “n” versions. I don’t quite
understand the reasoning behind the usage of each. If we guarantee that the
string is null terminated (and always null terminated) then we don’t need the
“n” versions. But even if we guarantee that the input string is null
terminated, it may accidentally have the null character chopped off by an off
by 1 error in a strdup or something like that. So my preference here would be
to use the “n” versions of all of the string functions. Thoughts?
{noformat}
command is guaranteed to be null terminated by the confiugration functions. If
we use the 'n' versions of the function we end up doing a 'begins with' match
instead of an exact match which could cause problems(e.g. "inspect" would match
"inspectcommand")
{noformat}
docker-util.c:read_and_verify_command_file()
+ if (current_len + string_len < bufflen - 1) {
+strncpy(buff + current_len, string, bufflen - current_len);
+
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135245#comment-16135245
]
Varun Vasudev commented on YARN-6623:
-
{quote}
#define EXECUTOR_PATH_MAX 131072
This is lots of allocation. The OS actually needs to zero all the allocated
memory before giving it to other processes after close, so this might add to
the overall CPU usage and memory bandwidth.
Given that yarn local dirs are bind mounted into containers, using 4096 has
proven problematic with even a small number of yarn local dirs/disks (3 in the
case I saw). We're going to need to make the size of this buffer configurable.
If it seems more appropriate to do so in a separate issue, I'm fine with that.
{quote}
This got fixed by YARN-6988.
> Add support to turn off launching privileged containers in the
> container-executor
> -
>
> Key: YARN-6623
> URL: https://issues.apache.org/jira/browse/YARN-6623
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
>Priority: Blocker
> Attachments: YARN-6623.001.patch, YARN-6623.002.patch,
> YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch
>
>
> Currently, launching privileged containers is controlled by the NM. We should
> add a flag to the container-executor.cfg allowing admins to disable launching
> privileged containers at the container-executor level.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-6623: Attachment: YARN-6623.005.patch New patch uploaded after rebasing to trunk. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Vasudev updated YARN-6623:
Attachment: YARN-6623.004.patch
Thanks for the review [~miklos.szeg...@cloudera.com]!
{quote}
/proc/mounts,/sys/fs/cgroup are always in the same place
This is actually not completely true. If you run in a container,/sys/fs/cgroup
can be anywhere
490.addReadOnlyMountLocation(CGROUPS_ROOT_DIRECTORY,
491CGROUPS_ROOT_DIRECTORY, false);
This is actually a security issue. As opposed to lxcfs, mounting cgroups will
expose information about all the other containers to each container. This
change is big enough but you might want to whitelist this in the future.
{quote}
This was already being set before this patch. Perhaps, it can be handled
cleanly as part of YARN-5534?
{quote}
SET(CMAKE_CXX_FLAGS "$\{CMAKE_CXX_FLAGS} -Wall -std=c++11")
Not sure, but this might not build on old OS like centos6
{quote}
Fixed.
{quote}
76 printWriter.println("\[docker-command-execution]");
77 for (Map.Entry entry :
cmd.getCommandArguments()
78 .entrySet()) \{
79printWriter.println(" " + entry.getKey() + "=" + StringUtils
80.join(",", entry.getValue()));
81 }
Probably it does worth to check, if entry for example contains “abc=\ndef” to
avoid injection attacks.
{quote}
Fixed.
{quote}
169 ret\[i + 1] = '\0';
I think it is safe to do a single ret\[I] = 0; after the loop
{quote}
Fixed.
{quote}
177void quote_and_append_arg(char **str, size_t *size, const char* param,
const char *arg) \{
178 char *tmp = escape_single_quote(arg);
179 strcat(*str, param);
180 strcat(*str, "'");
181 if(strlen(*str) + strlen(tmp) > *size) \{
182*str = (char *) realloc(*str, strlen(*str) + strlen(tmp) + 1024);
183if(*str == NULL) \{
184 exit(OUT_OF_MEMORY);
185}
186*size = strlen(*str) + strlen(tmp) + 1024;
187 }
188 strcat(*str, tmp);
189 strcat(*str, "' ");
190 free(tmp);
191}
What is 1024? How about setting * size before *str, so that there is no need
for duplication?
{quote}
Fixed.
{quote}
#define EXECUTOR_PATH_MAX 131072
This is lots of allocation. The OS actually needs to zero all the allocated
memory before giving it to other processes after close, so this might add to
the overall CPU usage and memory bandwidth.
{quote}
Fixed. Set back to 4096.
{quote}
35if (command != NULL) \{
36 free(command);
37}
This is not necessary, free always ignores NULL argument.
{quote}
Fixed.
{quote}
47 if (current_len + string_len < bufflen) \{
bufflen-1 to allocate space for the terminating 0
{quote}
Fixed.
{quote}
48strncpy(buff + current_len, string, bufflen - current_len);
strncpy does not add 0 terminator at the end of the target, if
strlen(string)==bufflen - current_len resulting in a read buffer overflow later.
{quote}
Fixed.
{quote}
165 if (0 == strncmp(container_name, CONTAINER_NAME_PREFIX,
strlen(CONTAINER_NAME_PREFIX))) \{
This is a double scan. You can just use strcmp with the same effect.
{quote}
{quote}
249 const char *regex_str =
"^((\[a-zA-Z0-9.-]+)(:\[0-9]+)?/)?(\[a-z0-9_./-]+)(:\[a-zA-Z0-9_.-]+)?$”;
Image can be specified by a digest, which is more secure. I do not see that
supported by the regex. IMAGE\[:TAG|@DIGEST]
{quote}
[~shaneku...@gmail.com] added this, I'm just moving it to a different file.
Shane can you please comment?
{quote}
# docker.binary=/bin/docker
115 docker_binary = strdup("/usr/bin/docker”);
We should choose one and use it everywhere.
{quote}
Fixed.
{quote}
477permitted_mount_len = strlen(permitted_mounts\[i]);
478if (permitted_mounts\[i]\[permitted_mount_len - 1] == '/') \{
Buffer underflow at permitted_mount_len == 0
{quote}
Fixed.
{quote}
488static char* normalize_mount(const char* mount) \{
It would be really nice to have some comments here.
{quote}
Fixed.
{quote}
503 size_t len = strlen(real_mount);
504 if (real_mount\[len - 1] != '/') {
505ret_ptr = (char *) alloc_and_clear_memory(len + 2, sizeof(char));
506strncpy(ret_ptr, real_mount, len);
507ret_ptr\[len] = '/';
508ret_ptr\[len + 1] = '\0';
Potential buffer underflow moreover most likely the character does not match
and we end with a normalized mount path of “/“. This happens, when
strlen(real_mount)==0
{quote}
Fixed
> Add support to turn off launching privileged containers in the
> container-executor
> -
>
> Key: YARN-6623
> URL: https://issues.apache.org/jira/browse/YARN-6623
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components:
[jira] [Commented] (YARN-6804) Allow custom hostname for docker containers in native services
[ https://issues.apache.org/jira/browse/YARN-6804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128425#comment-16128425 ] Varun Vasudev commented on YARN-6804: - I would like to get this into 2.8 as well please. There's some Docker work that needs to go into 2.8 that is blocked by this. Where are the conflicts? I can help resolve them. > Allow custom hostname for docker containers in native services > -- > > Key: YARN-6804 > URL: https://issues.apache.org/jira/browse/YARN-6804 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-native-services >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi > Fix For: 2.9.0, yarn-native-services, 3.0.0-beta1 > > Attachments: YARN-6804-trunk.004.patch, YARN-6804-trunk.005.patch, > YARN-6804-yarn-native-services.001.patch, > YARN-6804-yarn-native-services.002.patch, > YARN-6804-yarn-native-services.003.patch, > YARN-6804-yarn-native-services.004.patch, > YARN-6804-yarn-native-services.005.patch > > > Instead of the default random docker container hostname, we could set a more > user-friendly hostname for the container. The default could be a hostname > based on the container ID, with an option for the AM to provide a different > hostname. In the case of the native services AM, we could provide the > hostname that would be created by the registry DNS server. Regardless of > whether or not registry DNS is enabled, this would be a more useful hostname > for the docker container. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-6623: Attachment: YARN-6623.003.patch Uploaded 003.patch which fixes the failing cetest tests and has some checkstyle fixes. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6804) Allow custom hostname for docker containers in native services
[ https://issues.apache.org/jira/browse/YARN-6804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128329#comment-16128329 ] Varun Vasudev commented on YARN-6804: - [~jianhe] - can you backport the NM pieces to branch-2 and branch-2.8? It's blocking a bunch of other backports. Thanks! cc [~shaneku...@gmail.com], [~leftnoteasy] > Allow custom hostname for docker containers in native services > -- > > Key: YARN-6804 > URL: https://issues.apache.org/jira/browse/YARN-6804 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-native-services >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi > Fix For: yarn-native-services, 3.0.0-beta1 > > Attachments: YARN-6804-trunk.004.patch, YARN-6804-trunk.005.patch, > YARN-6804-yarn-native-services.001.patch, > YARN-6804-yarn-native-services.002.patch, > YARN-6804-yarn-native-services.003.patch, > YARN-6804-yarn-native-services.004.patch, > YARN-6804-yarn-native-services.005.patch > > > Instead of the default random docker container hostname, we could set a more > user-friendly hostname for the container. The default could be a hostname > based on the container ID, with an option for the AM to provide a different > hostname. In the case of the native services AM, we could provide the > hostname that would be created by the registry DNS server. Regardless of > whether or not registry DNS is enabled, this would be a more useful hostname > for the docker container. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6988) container-executor fails for docker when command length > 4096 B
[
https://issues.apache.org/jira/browse/YARN-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128325#comment-16128325
]
Varun Vasudev commented on YARN-6988:
-
Sounds good [~ebadger].
> container-executor fails for docker when command length > 4096 B
>
>
> Key: YARN-6988
> URL: https://issues.apache.org/jira/browse/YARN-6988
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: YARN-6988.001.patch
>
>
> {{run_docker}} and {{launch_docker_container_as_user}} allocate their command
> arrays using EXECUTOR_PATH_MAX, which is hardcoded to 4096 in
> configuration.h. Because of this, the full docker command can only be 4096
> characters. If it is longer, it will be truncated and the command will fail
> with a parsing error. Because of the bind-mounting of volumes, the arguments
> to the docker command can quickly get large. For example, I passed the 4096
> limit with an 11 disk node.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-6623: Attachment: YARN-6623.002.patch Uploaded a new patch to fix the test failures and findbugs warnings. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6988) container-executor fails for docker when command length > 4096 B
[
https://issues.apache.org/jira/browse/YARN-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16125926#comment-16125926
]
Varun Vasudev commented on YARN-6988:
-
[~ebadger] - I'm increasing the EXECUTOR_PATH_MAX to 128K as part of YARN-6623.
Can you please take a look?
> container-executor fails for docker when command length > 4096 B
>
>
> Key: YARN-6988
> URL: https://issues.apache.org/jira/browse/YARN-6988
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Eric Badger
>Assignee: Eric Badger
>
> {{run_docker}} and {{launch_docker_container_as_user}} allocate their command
> arrays using EXECUTOR_PATH_MAX, which is hardcoded to 4096 in
> configuration.h. Because of this, the full docker command can only be 4096
> characters. If it is longer, it will be truncated and the command will fail
> with a parsing error. Because of the bind-mounting of volumes, the arguments
> to the docker command can quickly get large. For example, I passed the 4096
> limit with an 11 disk node.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-6623: Attachment: YARN-6623.001.patch First cut of the patch. It's big because it moves a bunch of functionality into the container-executor and the subsequent refactor on the java code has a lot of changes. At a high level, the changes are - 1. Move the Docker command construction into the container-executor(docker-util.c) 2. The NM passes a file with params to the container-executor instead of a file containing the commands. 3. Added controls in container-executor.cfg to explicitly allow capabilities, mount directories, mount devices, and allowed networks. By default, nothing is allowed. 4. Added a control in container-executor to disable privileged containers. [~shaneku...@gmail.com], [~miklos.szeg...@cloudera.com], [~ebadger], [~leftnoteasy] - can you please take a look? Thanks! > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6789) new api to get all supported resources from RM
[ https://issues.apache.org/jira/browse/YARN-6789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123385#comment-16123385 ] Varun Vasudev commented on YARN-6789: - Strongly disagree with not having units. Today if a cluster wants to do scheduling in milliVcores, it requires updates to the RM, the NM and breaks *every* job that runs on the cluster because every job must be re-configured to run in milliVcores. Kubernetes has had support for units from the beginning, I don't see why YARN shouldn't have support for units as well. > new api to get all supported resources from RM > -- > > Key: YARN-6789 > URL: https://issues.apache.org/jira/browse/YARN-6789 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-6789-YARN-3926.001.patch > > > It will be better to provide an api to get all supported resource types from > RM. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5534) Allow whitelisted volume mounts
[ https://issues.apache.org/jira/browse/YARN-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119924#comment-16119924 ] Varun Vasudev commented on YARN-5534: - It's going to end up being a combination. Some settings have to be done in the container-executor.cfg(like whitelisted volume mounts), and some will go into yarn-site.xml. For example(just made up), an admin may want to mount /data-volume into every container by some subset of users. container-executor.cfg should have a setting permitting the mounting of /data-volume but yarn-site.xml should have feature to mount it into every container for those users. Does that make sense? > Allow whitelisted volume mounts > > > Key: YARN-5534 > URL: https://issues.apache.org/jira/browse/YARN-5534 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: luhuichun >Assignee: Shane Kumpf > Attachments: YARN-5534.001.patch, YARN-5534.002.patch, > YARN-5534.003.patch > > > Introduction > Mounting files or directories from the host is one way of passing > configuration and other information into a docker container. > We could allow the user to set a list of mounts in the environment of > ContainerLaunchContext (e.g. /dir1:/targetdir1,/dir2:/targetdir2). > These would be mounted read-only to the specified target locations. This has > been resolved in YARN-4595 > 2.Problem Definition > Bug mounting arbitrary volumes into a Docker container can be a security risk. > 3.Possible solutions > one approach to provide safe mounts is to allow the cluster administrator to > configure a set of parent directories as white list mounting directories. > Add a property named yarn.nodemanager.volume-mounts.white-list, when > container executor do mount checking, only the allowed directories or > sub-directories can be mounted. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16116129#comment-16116129 ] Varun Vasudev commented on YARN-6623: - To fix this, we need support for sections provided by YARN-6033. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4081) Add support for multiple resource types in the Resource class
[ https://issues.apache.org/jira/browse/YARN-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112305#comment-16112305 ] Varun Vasudev commented on YARN-4081: - The reason I added units was primarily to allow the NMs, the RMs and the user to differ in the way they specify units. For example, we have people requesting us to specify cpu in milliVCores. Today, this means that all the NMs must specify the resources in milliVCores, but there's no way to communicate this an admin or a user. Everyone(users + admins) must agree on the units being used. With the addition of units, we can set the RM to do all its computations in milliVCores, let the NMs specify the VCores in same format as today and the user can request for resources to be allocated in the unit they prefer. If you look at ResoureUtils#getNodeResourceInformation, it actually doesn't read the resource-types.xml file, it only reads the node-resources.xml file. Similarly, we don't assume that the user has access to the resource-types.xml file being used by the RM. They may have drifted apart, it shouldn't matter. Specifically, for the NMs, what happens behind the scenes is that the ClusterNodeTracker when it does the Resources.addTo in addNode, converts the NM's resource specified in it's own units to the units that the RM is dealing in and the math just flows through. Does that make sense? > Add support for multiple resource types in the Resource class > - > > Key: YARN-4081 > URL: https://issues.apache.org/jira/browse/YARN-4081 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: YARN-3926 > > Attachments: YARN-4081-YARN-3926.001.patch, > YARN-4081-YARN-3926.002.patch, YARN-4081-YARN-3926.003.patch, > YARN-4081-YARN-3926.004.patch, YARN-4081-YARN-3926.005.patch, > YARN-4081-YARN-3926.006.patch, YARN-4081-YARN-3926.007.patch, > YARN-4081-YARN-3926.008.patch > > > For adding support for multiple resource types, we need to add support for > this in the Resource class. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112271#comment-16112271 ] Varun Vasudev commented on YARN-6623: - Just to note - we should document this in the release notes since it'll break backwards compatibility(if we turn off all runtime except the process runtime by default). > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6033) Add support for sections in container-executor configuration file
[
https://issues.apache.org/jira/browse/YARN-6033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112257#comment-16112257
]
Varun Vasudev commented on YARN-6033:
-
{quote}
How about the name? I think we should should call trim on it in
get_section_name. Note: Our experience is that lots of users leave spaces
indentation here or there, which is understandable, since all other
configuration is XML.
{quote}
I can understand trimming the entries because folks may want to have something
like
{noformat}
[section-name]
key1=value1
key2=value2
{noformat}
but I'm not convinced about trimming the section name.
{noformat}
I found one more thing:
struct configuration CFG = {.size=0, .sections=NULL};
I believe this pattern is standard C++0x, the file is standard C
(container-executor.c).
{noformat}
This is valid as per the C99 standard - https://stackoverflow.com/a/330867
> Add support for sections in container-executor configuration file
> -
>
> Key: YARN-6033
> URL: https://issues.apache.org/jira/browse/YARN-6033
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Attachments: YARN-6033.003.patch, YARN-6033.004.patch,
> YARN-6033.005.patch, YARN-6033.006.patch, YARN-6033.007.patch,
> YARN-6033-YARN-5673.001.patch, YARN-6033-YARN-5673.002.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5534) Allow whitelisted volume mounts
[ https://issues.apache.org/jira/browse/YARN-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110834#comment-16110834 ] Varun Vasudev commented on YARN-5534: - bq. Thank you for the patch Shane Kumpf. Quick question, should not white-list-volume-mounts be a setting in container-executor.cfg instead of yarn-site.xml? Ideally it should be but the the current NM->container-exectuor interface doesn't allow for it. Once YARN-6033 is committed, I plan to rewrite it to do invocations via a config file and we can add the checks into the container-executor.cfg. > Allow whitelisted volume mounts > > > Key: YARN-5534 > URL: https://issues.apache.org/jira/browse/YARN-5534 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: luhuichun >Assignee: Shane Kumpf > Attachments: YARN-5534.001.patch, YARN-5534.002.patch, > YARN-5534.003.patch > > > Introduction > Mounting files or directories from the host is one way of passing > configuration and other information into a docker container. > We could allow the user to set a list of mounts in the environment of > ContainerLaunchContext (e.g. /dir1:/targetdir1,/dir2:/targetdir2). > These would be mounted read-only to the specified target locations. This has > been resolved in YARN-4595 > 2.Problem Definition > Bug mounting arbitrary volumes into a Docker container can be a security risk. > 3.Possible solutions > one approach to provide safe mounts is to allow the cluster administrator to > configure a set of parent directories as white list mounting directories. > Add a property named yarn.nodemanager.volume-mounts.white-list, when > container executor do mount checking, only the allowed directories or > sub-directories can be mounted. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6033) Add support for sections in container-executor configuration file
[
https://issues.apache.org/jira/browse/YARN-6033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Vasudev updated YARN-6033:
Attachment: YARN-6033.007.patch
bq. is_section_start_line and get_section_name should handle leading and
trailing spaces, just like the current behavior of the configuration lines.
I would prefer not to do this, we shouldn't allow section starts to be placed
anywhere on the line, nor should we strip trailing spaces.
{noformat}
264 len = strlen(line);
265 buffer = (char *) calloc(len + 1, sizeof(char));
266 strncpy(buffer, line, len);
You copy the value into the buffer right after allocation. If you do calloc to
put the trailing \0 there, it might be faster to do that explicitly, because
this code fills the buffer twice.
{noformat}
Fair point, but I prefer to use calloc. The speed shouldn't be a big deal for
us.
{noformat}
269 //if no equals is found ignore this line, can be an empty line also
This is a little bit controversial. I would ignore a line trimmed to empty and
return a configuration error for anything else. It helps with debugging config
issues
{noformat}
Agreed, fixed.
{noformat}
306 if ((section->size + 1) % MAX_SIZE == 0) {
307section->kv_pairs = (struct kv_pair **) realloc(
308section->kv_pairs,
309sizeof(struct kv_pair *) * (MAX_SIZE + section->size));
Is not it a better practice to do the increase of the kv buffer before we add a
new item? It might happen that we have MAX_SIZE items, then we allocate
MAX_SIZE*2. That might also simplify the code since we do not need to allocate
in the allocate_section.
{noformat}
Good point, fixed.
{noformat}
316 if (section->kv_pairs[section->size]) {
I think this check is redundant
{noformat}
Yep, fixed.
{noformat}
366static void populate_section_fields(FILE *conf_file, struct section
*section) {
Just a note: you would probably save some memory and extra copies, if this was
implemented as a state machine.
{noformat}
Yeah, I figured for now I'd keep the logic simple.
{noformat}
376 if (section->name != NULL) {
377if (read_section_entry(line, section) < 0) {
378 fprintf(ERRORFILE, "Error parsing line %s", line);
379 exit(INVALID_CONFIG_FILE);
380}
381 }
This code needs an else exiting with an error (kv pair without section), or am
I missing something? In that case it might need to be commented.
{noformat}
Yep, I missed that - fixed.
{noformat}
294 if (equaltok == NULL) {
295free((void *) section->kv_pairs[section->size]->key);
296free((void *) section->kv_pairs[section->size]);
297return 2;
298 }
Just to be safe, I would set the freed pointers to NULL here.
{noformat}
Fixed.
{noformat}
478 fseek(conf_file, 0, SEEK_SET);
You might want to add a comment // populate any entries in the newer format
(sections)
{noformat}
Added the comment but removed the fseek because we don't actually need it.
{noformat}
421 * @param free_section free the second section if set
422 */
423static void merge_sections(struct section *section1, struct section
*section2, const int free_section) {
If free_section is set, who will free section->name? BTW free_section is always
1, do we really need the parameter?
{noformat}
Nice catch on the name being freed, fixed. I think at some point for some
testing scenarios we might need it. If you feel strongly about it, I can remove
it.
> Add support for sections in container-executor configuration file
> -
>
> Key: YARN-6033
> URL: https://issues.apache.org/jira/browse/YARN-6033
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Attachments: YARN-6033.003.patch, YARN-6033.004.patch,
> YARN-6033.005.patch, YARN-6033.006.patch, YARN-6033.007.patch,
> YARN-6033-YARN-5673.001.patch, YARN-6033-YARN-5673.002.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6033) Add support for sections in container-executor configuration file
[ https://issues.apache.org/jira/browse/YARN-6033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-6033: Attachment: YARN-6033.006.patch Uploaded a new patch to fix the compilation error. > Add support for sections in container-executor configuration file > - > > Key: YARN-6033 > URL: https://issues.apache.org/jira/browse/YARN-6033 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: YARN-6033.003.patch, YARN-6033.004.patch, > YARN-6033.005.patch, YARN-6033.006.patch, YARN-6033-YARN-5673.001.patch, > YARN-6033-YARN-5673.002.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6033) Add support for sections in container-executor configuration file
[ https://issues.apache.org/jira/browse/YARN-6033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-6033: Attachment: YARN-6033.005.patch Thanks for the feedback [~wangda]! 1) I've added a null check in read_executor_config to prevent the crash 2) I've added the macro to configuration.c to address the compiler warning 3) I've added support for sections in a compatible manner with the existing config file format so now you can just add sections to the current config file. Can you please review the latest patch? Thanks! > Add support for sections in container-executor configuration file > - > > Key: YARN-6033 > URL: https://issues.apache.org/jira/browse/YARN-6033 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: YARN-6033.003.patch, YARN-6033.004.patch, > YARN-6033.005.patch, YARN-6033-YARN-5673.001.patch, > YARN-6033-YARN-5673.002.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6805) NPE in LinuxContainerExecutor due to null PrivilegedOperationException exit code
[
https://issues.apache.org/jira/browse/YARN-6805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16085466#comment-16085466
]
Varun Vasudev commented on YARN-6805:
-
Doesn't look like the findbugs warnings are related to the patch. +1 from me.
> NPE in LinuxContainerExecutor due to null PrivilegedOperationException exit
> code
>
>
> Key: YARN-6805
> URL: https://issues.apache.org/jira/browse/YARN-6805
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.8.1
>Reporter: Jason Lowe
>Assignee: Jason Lowe
> Attachments: YARN-6805.001.patch
>
>
> The LinuxContainerExecutor contains a number of code snippets like this:
> {code}
> } catch (PrivilegedOperationException e) {
> int exitCode = e.getExitCode();
> {code}
> PrivilegedOperationException#getExitCode can return null if the operation was
> interrupted, so when the JVM does auto-unboxing on that last line it can NPE
> if there was no exit code.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6033) Add support for sections in container-executor configuration file
[ https://issues.apache.org/jira/browse/YARN-6033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-6033: Attachment: YARN-6033.004.patch Uploaded a new patch to fix whitespace issue. > Add support for sections in container-executor configuration file > - > > Key: YARN-6033 > URL: https://issues.apache.org/jira/browse/YARN-6033 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: YARN-6033.003.patch, YARN-6033.004.patch, > YARN-6033-YARN-5673.001.patch, YARN-6033-YARN-5673.002.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6033) Add support for sections in container-executor configuration file
[
https://issues.apache.org/jira/browse/YARN-6033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Vasudev updated YARN-6033:
Attachment: YARN-6033.003.patch
Thanks for the reviews [~sunil.gov...@gmail.com], [~leftnoteasy]!
bq. 1. one doubt. In trim_comment, strchr will get first occurrence of the
comment character rt, so it ll be replaced with a '/0'. What if two comment
chars are present in the start of line?
It shouldn't matter, we'll just skip the line.
{quote}
2.
merge_sections((struct section *) existing_section, new_section, 1);
In existing section, will it have enough memory to hold new section?
{quote}
Good catch, fixed.
{quote}
3.
struct section *new_section = read_section(conf_file);
In read_config, if below code returns null, we will still go and do realloc for
increase sectionentry details.
{quote}
Yeah, it shouldn't matter because we'll exit the loop no?
{quote}
1) It's better to move
struct section executor_cfg = \{.size=0, .sectiondetails=NULL};
and
struct configuration CFG = \{.size=0, .sections=NULL};
>From container-executor.c to configurations.c. And add getter/setter method to
>configuration.h. I think we should not couple life cycle of configuration and
>container-executor since we could add other modules beyond container-executor
>in the new design.
{quote}
I've avoided this for now. My thinking was that each module could just create
it's own version of the configuration object and manage the life cycle of the
object. What do you think?
{quote}
2) some rename suggestions:
sectionentry: is it better to call kv_pair?
sectiondetails: if you agree with above, how about rename it to kv_pairs?
{quote}
Fixed.
> Add support for sections in container-executor configuration file
> -
>
> Key: YARN-6033
> URL: https://issues.apache.org/jira/browse/YARN-6033
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Attachments: YARN-6033.003.patch, YARN-6033-YARN-5673.001.patch,
> YARN-6033-YARN-5673.002.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6761) Fix build for YARN-3926 branch
[ https://issues.apache.org/jira/browse/YARN-6761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072245#comment-16072245 ] Varun Vasudev commented on YARN-6761: - Since YARN-3926 is already broken, the pre-commit build the mvninstall steps, etc failed, as a result a lot of the subsequent steps show up as -1. Please note that with the patch applied to YARN-3926, the mvninstall and compile steps pass. > Fix build for YARN-3926 branch > -- > > Key: YARN-6761 > URL: https://issues.apache.org/jira/browse/YARN-6761 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: YARN-6761-YARN-3926.001.patch > > > After rebasing to trunk, due to the addition of YARN-6679, compilation of the > YARN-3926 branch is broken. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6761) Fix build for YARN-3926 branch
[ https://issues.apache.org/jira/browse/YARN-6761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-6761: Attachment: YARN-6761-YARN-3926.001.patch This patch addresses the build break. I've kept it simple for now, but if folks don't like the static Resource object created, we can address that by moving the ResourceUtils class to the hadoop-yarn-api module from the hadoop-yarn-common module(where it currently resides). [~templedf], [~sunil.gov...@gmail.com], [~leftnoteasy] - can you please take a look? Thanks! > Fix build for YARN-3926 branch > -- > > Key: YARN-6761 > URL: https://issues.apache.org/jira/browse/YARN-6761 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: YARN-6761-YARN-3926.001.patch > > > After rebasing to trunk, due to the addition of YARN-6679, compilation of the > YARN-3926 branch is broken. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6761) Fix build for YARN-3926 branch
Varun Vasudev created YARN-6761: --- Summary: Fix build for YARN-3926 branch Key: YARN-6761 URL: https://issues.apache.org/jira/browse/YARN-6761 Project: Hadoop YARN Issue Type: Sub-task Reporter: Varun Vasudev Assignee: Varun Vasudev After rebasing to trunk, due to the addition of YARN-6679, compilation of the YARN-3926 branch is broken. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6630) Container worker dir could not recover when NM restart
[
https://issues.apache.org/jira/browse/YARN-6630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045900#comment-16045900
]
Varun Vasudev commented on YARN-6630:
-
Assigned issue to [~fly_in_gis] and kicked off another Jenkins run because the
earlier links aren't accessible anymore.
> Container worker dir could not recover when NM restart
> --
>
> Key: YARN-6630
> URL: https://issues.apache.org/jira/browse/YARN-6630
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yang Wang
>Assignee: Yang Wang
> Attachments: YARN-6630.001.patch
>
>
> When ContainerRetryPolicy is NEVER_RETRY, container worker dir will not be
> saved in NM state store.
> {code:title=ContainerLaunch.java}
> ...
> private void recordContainerWorkDir(ContainerId containerId,
> String workDir) throws IOException{
> container.setWorkDir(workDir);
> if (container.isRetryContextSet()) {
> context.getNMStateStore().storeContainerWorkDir(containerId, workDir);
> }
> }
> {code}
> Then NM restarts, container.workDir could not recover and is null, and may
> cause some exceptions.
> We already have a problem, after NM restart, we send a resource localization
> request while container is running(YARN-1503), then NM will fail because of
> the following exception.
> So, container.workdir always need to be saved in NM state store.
> {code:title=ContainerImpl.java}
> static class ResourceLocalizedWhileRunningTransition
> extends ContainerTransition {
> ...
> String linkFile = new Path(container.workDir, link).toString();
> ...
> {code}
> {code}
> java.lang.IllegalArgumentException: Can not create a Path from a null string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:159)
> at org.apache.hadoop.fs.Path.(Path.java:175)
> at org.apache.hadoop.fs.Path.(Path.java:110)
> ... ...
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6630) Container worker dir could not recover when NM restart
[
https://issues.apache.org/jira/browse/YARN-6630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Vasudev reassigned YARN-6630:
---
Assignee: Yang Wang
> Container worker dir could not recover when NM restart
> --
>
> Key: YARN-6630
> URL: https://issues.apache.org/jira/browse/YARN-6630
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yang Wang
>Assignee: Yang Wang
> Attachments: YARN-6630.001.patch
>
>
> When ContainerRetryPolicy is NEVER_RETRY, container worker dir will not be
> saved in NM state store.
> {code:title=ContainerLaunch.java}
> ...
> private void recordContainerWorkDir(ContainerId containerId,
> String workDir) throws IOException{
> container.setWorkDir(workDir);
> if (container.isRetryContextSet()) {
> context.getNMStateStore().storeContainerWorkDir(containerId, workDir);
> }
> }
> {code}
> Then NM restarts, container.workDir could not recover and is null, and may
> cause some exceptions.
> We already have a problem, after NM restart, we send a resource localization
> request while container is running(YARN-1503), then NM will fail because of
> the following exception.
> So, container.workdir always need to be saved in NM state store.
> {code:title=ContainerImpl.java}
> static class ResourceLocalizedWhileRunningTransition
> extends ContainerTransition {
> ...
> String linkFile = new Path(container.workDir, link).toString();
> ...
> {code}
> {code}
> java.lang.IllegalArgumentException: Can not create a Path from a null string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:159)
> at org.apache.hadoop.fs.Path.(Path.java:175)
> at org.apache.hadoop.fs.Path.(Path.java:110)
> ... ...
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6366) Refactor the NodeManager DeletionService to support additional DeletionTask types.
[ https://issues.apache.org/jira/browse/YARN-6366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025911#comment-16025911 ] Varun Vasudev commented on YARN-6366: - [~shaneku...@gmail.com] - the patch doesn't compile cleanly on branch-2. Can you please take a look? Thanks! > Refactor the NodeManager DeletionService to support additional DeletionTask > types. > -- > > Key: YARN-6366 > URL: https://issues.apache.org/jira/browse/YARN-6366 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Attachments: YARN-6366.001.patch, YARN-6366.002.patch, > YARN-6366.003.patch, YARN-6366.004.patch, YARN-6366.005.patch, > YARN-6366.006.patch, YARN-6366.007.patch, YARN-6366.008.patch > > > The NodeManager's DeletionService only supports file based DeletionTask's. > This makes sense as files (and directories) have been the primary concern for > clean up to date. With the addition of the Docker container runtime, addition > types of DeletionTask are likely to be required, such as deletion of docker > container and images. See YARN-5366 and YARN-5670. This issue is to refactor > the DeletionService to support additional DeletionTask's. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6366) Refactor the NodeManager DeletionService to support additional DeletionTask types.
[ https://issues.apache.org/jira/browse/YARN-6366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019584#comment-16019584 ] Varun Vasudev commented on YARN-6366: - +1 for the latest patch. I'll commit this tomorrow if no one objects. > Refactor the NodeManager DeletionService to support additional DeletionTask > types. > -- > > Key: YARN-6366 > URL: https://issues.apache.org/jira/browse/YARN-6366 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Attachments: YARN-6366.001.patch, YARN-6366.002.patch, > YARN-6366.003.patch, YARN-6366.004.patch, YARN-6366.005.patch, > YARN-6366.006.patch, YARN-6366.007.patch, YARN-6366.008.patch > > > The NodeManager's DeletionService only supports file based DeletionTask's. > This makes sense as files (and directories) have been the primary concern for > clean up to date. With the addition of the Docker container runtime, addition > types of DeletionTask are likely to be required, such as deletion of docker > container and images. See YARN-5366 and YARN-5670. This issue is to refactor > the DeletionService to support additional DeletionTask's. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6622) Document Docker work as experimental
[ https://issues.apache.org/jira/browse/YARN-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16015582#comment-16015582 ] Varun Vasudev commented on YARN-6622: - [~chris.douglas], [~templedf], [~sidharta-s], [~shaneku...@gmail.com] - can you take a look and let me know if the modified documentation looks ok? > Document Docker work as experimental > > > Key: YARN-6622 > URL: https://issues.apache.org/jira/browse/YARN-6622 > Project: Hadoop YARN > Issue Type: Task > Components: documentation >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: YARN-6622.001.patch > > > We should update the Docker support documentation calling out the Docker work > as experimental. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6622) Document Docker work as experimental
[ https://issues.apache.org/jira/browse/YARN-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-6622: Attachment: YARN-6622.001.patch > Document Docker work as experimental > > > Key: YARN-6622 > URL: https://issues.apache.org/jira/browse/YARN-6622 > Project: Hadoop YARN > Issue Type: Task > Components: documentation >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: YARN-6622.001.patch > > > We should update the Docker support documentation calling out the Docker work > as experimental. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6622) Document Docker work as experimental
[ https://issues.apache.org/jira/browse/YARN-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-6622: Component/s: documentation > Document Docker work as experimental > > > Key: YARN-6622 > URL: https://issues.apache.org/jira/browse/YARN-6622 > Project: Hadoop YARN > Issue Type: Task > Components: documentation >Reporter: Varun Vasudev >Assignee: Varun Vasudev > > We should update the Docker support documentation calling out the Docker work > as experimental. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
Varun Vasudev created YARN-6623: --- Summary: Add support to turn off launching privileged containers in the container-executor Key: YARN-6623 URL: https://issues.apache.org/jira/browse/YARN-6623 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Reporter: Varun Vasudev Assignee: Varun Vasudev Currently, launching privileged containers is controlled by the NM. We should add a flag to the container-executor.cfg allowing admins to disable launching privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6622) Document Docker work as experimental
Varun Vasudev created YARN-6622: --- Summary: Document Docker work as experimental Key: YARN-6622 URL: https://issues.apache.org/jira/browse/YARN-6622 Project: Hadoop YARN Issue Type: Task Reporter: Varun Vasudev Assignee: Varun Vasudev We should update the Docker support documentation calling out the Docker work as experimental. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6366) Refactor the NodeManager DeletionService to support additional DeletionTask types.
[ https://issues.apache.org/jira/browse/YARN-6366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16006699#comment-16006699 ] Varun Vasudev commented on YARN-6366: - bq. Regarding #4) Moving the conversion methods into ProtoUtils creates a circular dependency between hadoop-common and the nodemanager. Would a consolidated NMProtoUtils within the nodemanager module suffice? Yep. Makes sense. bq. Regarding #5) If you take a look at the diff for that test, the existing test passes an empty Path array. I've tried to keep the test functionally the same. Got it. No need to change it then. > Refactor the NodeManager DeletionService to support additional DeletionTask > types. > -- > > Key: YARN-6366 > URL: https://issues.apache.org/jira/browse/YARN-6366 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Attachments: YARN-6366.001.patch, YARN-6366.002.patch, > YARN-6366.003.patch, YARN-6366.004.patch, YARN-6366.005.patch, > YARN-6366.006.patch > > > The NodeManager's DeletionService only supports file based DeletionTask's. > This makes sense as files (and directories) have been the primary concern for > clean up to date. With the addition of the Docker container runtime, addition > types of DeletionTask are likely to be required, such as deletion of docker > container and images. See YARN-5366 and YARN-5670. This issue is to refactor > the DeletionService to support additional DeletionTask's. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
