[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16451041#comment-16451041 ] Hudson commented on YARN-6623: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14057 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14057/]) YARN-6623. Add support to turn off launching privileged containers in (xyao: rev 96afa69716bf1685fb8f2776ef61edfa4cd3a77f) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/modules/common/module-configs.h * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/DockerPullCommand.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/TestDockerCommandExecutor.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/TestDockerLoadCommand.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test_util.cc * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.h * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/TestDockerRunCommand.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/DockerContainers.md * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test-string-utils.cc * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/DockerClient.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/get_executable.h * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/DockerStopCommand.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/TestDockerContainerRuntime.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/DockerCommand.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/get_executable.c * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/DockerLoadCommand.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/DockerInspectCommand.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/DockerRunCommand.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/modules/common/module-configs.c * (edit) hadoop-yarn-project/hadoop-yarn/conf/container-executor.cfg * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/DockerRmCommand.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16227526#comment-16227526 ] Andrew Wang commented on YARN-6623: --- Hi Varun, do you mind adding a release note for this JIRA, given that it's marked as incompatible? Thanks! > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Fix For: 2.9.0, 3.0.0 > > Attachments: YARN-6623-branch-2.013.patch, > YARN-6623-branch-2.014.patch, YARN-6623-branch-2.015.patch, > YARN-6623.001.patch, YARN-6623.002.patch, YARN-6623.003.patch, > YARN-6623.004.patch, YARN-6623.005.patch, YARN-6623.006.patch, > YARN-6623.007.patch, YARN-6623.008.patch, YARN-6623.009.patch, > YARN-6623.010.patch, YARN-6623.011.patch, YARN-6623.012.patch, > YARN-6623.013.patch, cetest.stderr, cetest.stdout > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16210188#comment-16210188 ] Wangda Tan commented on YARN-6623: -- Attached patch LGTM, thanks [~vvasudev]. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Fix For: 3.0.0 > > Attachments: YARN-6623-branch-2.013.patch, > YARN-6623-branch-2.014.patch, YARN-6623-branch-2.015.patch, > YARN-6623.001.patch, YARN-6623.002.patch, YARN-6623.003.patch, > YARN-6623.004.patch, YARN-6623.005.patch, YARN-6623.006.patch, > YARN-6623.007.patch, YARN-6623.008.patch, YARN-6623.009.patch, > YARN-6623.010.patch, YARN-6623.011.patch, YARN-6623.012.patch, > YARN-6623.013.patch, cetest.stderr, cetest.stdout > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16207833#comment-16207833 ] Eric Yang commented on YARN-6623: - [~vvasudev] I opened YARN-7344 to track failed unit test issues on CentOS7. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Fix For: 3.0.0 > > Attachments: YARN-6623-branch-2.013.patch, > YARN-6623-branch-2.014.patch, YARN-6623-branch-2.015.patch, > YARN-6623.001.patch, YARN-6623.002.patch, YARN-6623.003.patch, > YARN-6623.004.patch, YARN-6623.005.patch, YARN-6623.006.patch, > YARN-6623.007.patch, YARN-6623.008.patch, YARN-6623.009.patch, > YARN-6623.010.patch, YARN-6623.011.patch, YARN-6623.012.patch, > YARN-6623.013.patch, cetest.stderr, cetest.stdout > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16207265#comment-16207265 ] Varun Vasudev commented on YARN-6623: - [~leftnoteasy] - can you take a look at the branch-2 patch please? Thanks! > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Fix For: 3.0.0 > > Attachments: YARN-6623-branch-2.013.patch, > YARN-6623-branch-2.014.patch, YARN-6623-branch-2.015.patch, > YARN-6623.001.patch, YARN-6623.002.patch, YARN-6623.003.patch, > YARN-6623.004.patch, YARN-6623.005.patch, YARN-6623.006.patch, > YARN-6623.007.patch, YARN-6623.008.patch, YARN-6623.009.patch, > YARN-6623.010.patch, YARN-6623.011.patch, YARN-6623.012.patch, > YARN-6623.013.patch, cetest.stderr, cetest.stdout > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16207264#comment-16207264
]
Varun Vasudev commented on YARN-6623:
-
[~eyang] - the tests did run as part of the trunk pre-commit build. The test
reports are lost now but it will appear under the section (root) in the test
report.
The tests seem to pass for me on the Mac and an Ubuntu VM. What's the
environment you're using? Maybe you should open a new ticket with details?
Here's the result I got -
Mac
{noformat}
Varuns-MacBook-Pro:hadoop-yarn-server-nodemanager varun$ mvn test -Dtest=cetest
-Pnative
[INFO] Scanning for projects...
[INFO]
[INFO]
[INFO] Building Apache Hadoop YARN NodeManager 3.1.0-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @
hadoop-yarn-server-nodemanager ---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- hadoop-maven-plugins:3.1.0-SNAPSHOT:protoc (compile-protoc) @
hadoop-yarn-server-nodemanager ---
[INFO] No changes detected in protoc files, skipping generation.
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @
hadoop-yarn-server-nodemanager ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
hadoop-yarn-server-nodemanager ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 4 resources
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @
hadoop-yarn-server-nodemanager ---
[INFO] Compiling 3 source files to
/Users/varun/Workspace/apache/committer-rw/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/classes
[INFO]
[INFO] --- hadoop-maven-plugins:3.1.0-SNAPSHOT:cmake-compile (cmake-compile) @
hadoop-yarn-server-nodemanager ---
[INFO] Running cmake
/Users/varun/Workspace/apache/committer-rw/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src
-DHADOOP_CONF_DIR=../etc/hadoop -DJVM_ARCH_DATA_MODEL=64 -G Unix Makefiles
[INFO] with extra environment variables {
CFLAGS = ''
}
[INFO] Running make -j 8 VERBOSE=1
[INFO] cmake compilation finished successfully in 205 millisecond(s).
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (make) @ hadoop-yarn-server-nodemanager
---
[INFO] Executing tasks
main:
[copy] Copying 4 files to
/Users/varun/Workspace/apache/committer-rw/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/test
[INFO] Executed tasks
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @
hadoop-yarn-server-nodemanager ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 7 resources
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
hadoop-yarn-server-nodemanager ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-surefire-plugin:2.17:test (default-test) @
hadoop-yarn-server-nodemanager ---
[INFO]
[INFO] --- hadoop-maven-plugins:3.1.0-SNAPSHOT:cmake-test
(test-container-executor) @ hadoop-yarn-server-nodemanager ---
[INFO]
[INFO] --- hadoop-maven-plugins:3.1.0-SNAPSHOT:cmake-test (cetest) @
hadoop-yarn-server-nodemanager ---
[INFO] ---
[INFO] C M A K E B U I L D E RT E S T
[INFO] ---
[INFO] cetest: running
/Users/varun/Workspace/apache/committer-rw/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/test/cetest
--gtest_filter=-Perf.
--gtest_output=xml:/Users/varun/Workspace/apache/committer-rw/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/surefire-reports/TEST-cetest.xml
[INFO] with extra environment variables {}
[INFO] STATUS: SUCCESS after 87 millisecond(s).
[INFO] ---
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 4.263 s
[INFO] Finished at: 2017-10-17T15:43:32+05:30
[INFO] Final Memory: 35M/318M
[INFO]
{noformat}
Ubuntu 16
{noformat}
varun@nm-u16:~/Workspace/apache/dev/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager$
mvn test -Dtest=cetest -Pnative
[INFO] Scanning for projects...
[INFO]
[INFO]
[INFO] Building Apache Hadoop YARN NodeManager 3.1.0-SNAPSHOT
[INFO] ---
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206764#comment-16206764
]
Eric Yang commented on YARN-6623:
-
The current trunk seems to fail on these test cases:
{code}
[ FAILED ] 3 tests, listed below:
[ FAILED ] TestDockerUtil.test_check_mount_permitted
[ FAILED ] TestDockerUtil.test_normalize_mounts
[ FAILED ] TestDockerUtil.test_add_rw_mounts
{code}
> Add support to turn off launching privileged containers in the
> container-executor
> -
>
> Key: YARN-6623
> URL: https://issues.apache.org/jira/browse/YARN-6623
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
>Priority: Blocker
> Fix For: 3.0.0
>
> Attachments: YARN-6623-branch-2.013.patch,
> YARN-6623-branch-2.014.patch, YARN-6623-branch-2.015.patch,
> YARN-6623.001.patch, YARN-6623.002.patch, YARN-6623.003.patch,
> YARN-6623.004.patch, YARN-6623.005.patch, YARN-6623.006.patch,
> YARN-6623.007.patch, YARN-6623.008.patch, YARN-6623.009.patch,
> YARN-6623.010.patch, YARN-6623.011.patch, YARN-6623.012.patch,
> YARN-6623.013.patch
>
>
> Currently, launching privileged containers is controlled by the NM. We should
> add a flag to the container-executor.cfg allowing admins to disable launching
> privileged containers at the container-executor level.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206761#comment-16206761
]
Eric Yang commented on YARN-6623:
-
Talked to [~wangda] and he said the test can be triggered using:
{code}
-Dtest=cetest,test-container-executor -Pnative
{code}
To the maven command. It would be nice if it is covered by default test
without passing in test case name.
> Add support to turn off launching privileged containers in the
> container-executor
> -
>
> Key: YARN-6623
> URL: https://issues.apache.org/jira/browse/YARN-6623
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
>Priority: Blocker
> Fix For: 3.0.0
>
> Attachments: YARN-6623-branch-2.013.patch,
> YARN-6623-branch-2.014.patch, YARN-6623-branch-2.015.patch,
> YARN-6623.001.patch, YARN-6623.002.patch, YARN-6623.003.patch,
> YARN-6623.004.patch, YARN-6623.005.patch, YARN-6623.006.patch,
> YARN-6623.007.patch, YARN-6623.008.patch, YARN-6623.009.patch,
> YARN-6623.010.patch, YARN-6623.011.patch, YARN-6623.012.patch,
> YARN-6623.013.patch
>
>
> Currently, launching privileged containers is controlled by the NM. We should
> add a flag to the container-executor.cfg allowing admins to disable launching
> privileged containers at the container-executor level.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206736#comment-16206736 ] Eric Yang commented on YARN-6623: - [~vvasudev] How is the C++ test case getting triggered? mvn clean test -Pnative doesn't trigger test_docker_util.cc to run. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Fix For: 3.0.0 > > Attachments: YARN-6623-branch-2.013.patch, > YARN-6623-branch-2.014.patch, YARN-6623-branch-2.015.patch, > YARN-6623.001.patch, YARN-6623.002.patch, YARN-6623.003.patch, > YARN-6623.004.patch, YARN-6623.005.patch, YARN-6623.006.patch, > YARN-6623.007.patch, YARN-6623.008.patch, YARN-6623.009.patch, > YARN-6623.010.patch, YARN-6623.011.patch, YARN-6623.012.patch, > YARN-6623.013.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206505#comment-16206505
]
Hadoop QA commented on YARN-6623:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 17m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 13 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
49s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
47s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
53s{color} | {color:green} branch-2 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
59s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
22s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m
49s{color} | {color:green} hadoop-yarn-project_hadoop-yarn generated 0 new + 0
unchanged - 1 fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated
0 new + 24 unchanged - 5 fixed = 24 total (was 29) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}104m 13s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 13m 58s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
7s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
21s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}176m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.scheduler.TestDistributedScheduler |
| | hadoop.yarn.server.nodemanager.scheduler.TestDistributedScheduler |
| Timed out junit tests |
org.apache.hadoop.yarn.server.nodemanager.TestNodeStatusUpdater |
| | org.apache.hadoop.yarn.server.resourcemanager.TestRM
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206061#comment-16206061
]
Hadoop QA commented on YARN-6623:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 13 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
2s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
40s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
47s{color} | {color:green} branch-2 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
55s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
16s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 1m
48s{color} | {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
25s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m
29s{color} | {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 1m 29s{color} |
{color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 29s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated
0 new + 24 unchanged - 5 fixed = 24 total (was 29) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 1m
22s{color} | {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
26s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
17s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 55m 46s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 44s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
9s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}102m 21s{color} |
{color:black} {color} |
\\
\\
|| Reas
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16205697#comment-16205697
]
Hadoop QA commented on YARN-6623:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 13 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
54s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
36s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
35s{color} | {color:green} branch-2 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
52s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
8s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 1m
44s{color} | {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
20s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m
13s{color} | {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 1m 13s{color} |
{color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 13s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated
0 new + 24 unchanged - 5 fixed = 24 total (was 29) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 1m
14s{color} | {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
18s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
15s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 17m 45s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 18s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
7s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 51m 27s{color} |
{color:black} {color} |
\\
\\
|| Reas
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16193974#comment-16193974 ] Eric Yang commented on YARN-6623: - [~ebadger] In some secure environment, I have seen that container-executor.cfg is set to non-world readable because the administrator doesn't want people to know about the allowed and bannded users on the cluster. Another possibility is the default umask are set to 027, when admin generates container-executor.cfg via Ambari. it is often set to world non-readable for secure environment. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Fix For: 3.0.0 > > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch, YARN-6623.011.patch, > YARN-6623.012.patch, YARN-6623.013.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16193001#comment-16193001 ] Eric Badger commented on YARN-6623: --- bq. It can be deprecated or used for fast-failing containers. I'm open to either option. Is there a reason we can't make container-executor.cfg world readable so that we can parse it in the NM before the container-executor is launched and fail fast that way? > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Fix For: 3.0.0 > > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch, YARN-6623.011.patch, > YARN-6623.012.patch, YARN-6623.013.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16192875#comment-16192875
]
Varun Vasudev commented on YARN-6623:
-
{quote}
There's already a config in yarn-site.xml
"yarn.nodemanager.runtime.linux.docker.allowed-container-networks"
Will the one in yarn-site.xml be deprecated ? what's the story here?
{quote}
It can be deprecated or used for fast-failing containers. I'm open to either
option.
bq. Also, should the default value include host and bridge like the
yarn-defalut.xml did ? otherwise, the default setting will just not work, user
has to explicitly chage it
No, the idea is to have admins explicitly change it.
> Add support to turn off launching privileged containers in the
> container-executor
> -
>
> Key: YARN-6623
> URL: https://issues.apache.org/jira/browse/YARN-6623
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
>Priority: Blocker
> Fix For: 3.0.0
>
> Attachments: YARN-6623.001.patch, YARN-6623.002.patch,
> YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch,
> YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch,
> YARN-6623.009.patch, YARN-6623.010.patch, YARN-6623.011.patch,
> YARN-6623.012.patch, YARN-6623.013.patch
>
>
> Currently, launching privileged containers is controlled by the NM. We should
> add a flag to the container-executor.cfg allowing admins to disable launching
> privileged containers at the container-executor level.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16192537#comment-16192537 ] Jian He commented on YARN-6623: --- Got a question, for this config added docker.allowed.networks=## comma seperated networks that can be used. e.g bridge,host,none There's already a config in yarn-site.xml "yarn.nodemanager.runtime.linux.docker.allowed-container-networks" Will the one in yarn-site.xml be deprecated ? what's the story here? Also, should the default value include host and bridge like the yarn-defalut.xml did ? otherwise, the default setting will just not work, user has to explicitly chage it > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Fix For: 3.0.0 > > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch, YARN-6623.011.patch, > YARN-6623.012.patch, YARN-6623.013.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16186186#comment-16186186 ] Andrew Wang commented on YARN-6623: --- Sorry, this missed the train for RC0. We'll pick it up for GA, or possibly if there's an RC1. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Fix For: 3.0.0 > > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch, YARN-6623.011.patch, > YARN-6623.012.patch, YARN-6623.013.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16185866#comment-16185866 ] Varun Vasudev commented on YARN-6623: - Thanks for the reviews [~ebadger], [~miklos.szeg...@cloudera.com], [~shaneku...@gmail.com], [~sunil.gov...@gmail.com] and [~leftnoteasy]! I'll get a patch together for branch-2. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Fix For: 3.0.0-beta1 > > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch, YARN-6623.011.patch, > YARN-6623.012.patch, YARN-6623.013.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16185134#comment-16185134 ] Hudson commented on YARN-6623: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12994 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/12994/]) YARN-6623. Add support to turn off launching privileged containers in (wangda: rev d3b1c6319546706c41a2011ead6c3fe208883200) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/TestDockerStopCommand.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/DockerRmCommand.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/DockerRunCommand.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/TestDockerRunCommand.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test_util.cc * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/TestDockerContainerRuntime.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/DockerCommandExecutor.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.h * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/TestDockerCommandExecutor.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test-string-utils.cc * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/TestDockerLoadCommand.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.h * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/util.h * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/TestDockerInspectCommand.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/util.c * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/get_executable.h * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/resources/test/test-configurations/docker-container-executor.cfg * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/TestDockerRmCommand.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/DockerLoadCommand.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/string-utils.c * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DockerLinuxContainerRuntime.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanage
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16185131#comment-16185131 ] Wangda Tan commented on YARN-6623: -- [~andrew.wang], I just saw your email for branch-3.0.0-beta1 cut, is it still open for new commits? Can we merge this to branch-3.0.0-beta1? > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Fix For: 3.0.0-beta1 > > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch, YARN-6623.011.patch, > YARN-6623.012.patch, YARN-6623.013.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16185129#comment-16185129 ] Wangda Tan commented on YARN-6623: -- Discussed with [~eyang] / [~vvasudev] / [~shaneku...@gmail.com], [~eyang] mentioned that we can move the whitelist usability issue to a separate JIRA to unblock this one. So I just pushed to trunk/branch-3.0, it has some conflicts to branch-2. Thanks to [~vvasudev] for the hard work and thanks [~miklos.szeg...@cloudera.com] / [~ebadger] / [~shaneku...@gmail.com] / [~eyang] / [~templedf] / [~djp] / [~andrew.wang] for reviews and suggestions! > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Fix For: 3.0.0-beta1 > > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch, YARN-6623.011.patch, > YARN-6623.012.patch, YARN-6623.013.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16184433#comment-16184433 ] Andrew Wang commented on YARN-6623: --- Thanks Varun. Considering this is targeted at 2.8.2, if we're okay taking that incompatibility for a maintenance release, I'm okay taking it between beta1 and GA. I'll update the target version, though if beta1 drags on, we can try and squeeze this in. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch, YARN-6623.011.patch, > YARN-6623.012.patch, YARN-6623.013.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16184415#comment-16184415 ] Miklos Szegedi commented on YARN-6623: -- If the decision is to ignore this bug for beta1, how about renaming the switch feature.docker.enabled to feature.docker.experimental.enabled or feature.docker.unsecure.enabled to make the decision clear to users? > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch, YARN-6623.011.patch, > YARN-6623.012.patch, YARN-6623.013.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16184174#comment-16184174 ] Shane Kumpf commented on YARN-6623: --- Thanks for addressing my comments. I can confirm that docker inspect is now functioning as expected. Sorry for the delay on this update, I was running into an issue with tc/network as a resource with this patch applied, however, I have been able to confirm that this is an existing issue that is unrelated to this patch. +1 (non-binding) on the latest patch. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch, YARN-6623.011.patch, > YARN-6623.012.patch, YARN-6623.013.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16183909#comment-16183909 ] Varun Vasudev commented on YARN-6623: - [~andrew.wang] - the one thing you should be aware of is that YARN-6623 will break backwards compatibility because from earlier releases because it disables most Docker features by default. So folks upgrading from beta-1 to beta-2 will find their Docker containers will no longer work. If you're fine with that, please go ahead with beta-1. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch, YARN-6623.011.patch, > YARN-6623.012.patch, YARN-6623.013.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16183050#comment-16183050 ] Andrew Wang commented on YARN-6623: --- Folks, this is currently marked as the last blocker for beta1. I asked Mikos about this offline, and he explained to me that: * We've documented the Docker feature as experimental * It's off by default * This isn't a regression from an earlier 3.0.0 alpha release Given this, I'd like to drop this off the blocker list and proceed with the beta1 release. Alternatively, if there's some quick hack we could do to unblock the release, I'm all ears. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch, YARN-6623.011.patch, > YARN-6623.012.patch, YARN-6623.013.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16181845#comment-16181845 ] Eric Yang commented on YARN-6623: - [~wangda] How does a non-privileged user acquire excessive permission by executing c-e? root:yarn is typically the owner of c-e binary. The user has to be root or yarn to run the binary. Hence, validation done by YARN user would be better than doing post validation after root privilege is acquired. One can argue that YARN user does not have access to check mount points, hence the validation needs to happen at root user level. If docker container is started for unprivileged user by using -u [uid]:[gid], Linux file system ACL still applies to process inside container. There will be no extra permission gain with mounting unauthorized path. In the previous implementation, there was no effective group id passed to docker. This was the reason that extra permission was gain through effective group. When this security hole is closed by YARN-4266, then there is no gain to shift validation logic to root user side for mount point permission validation. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch, YARN-6623.011.patch, > YARN-6623.012.patch, YARN-6623.013.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16181628#comment-16181628
]
Wangda Tan commented on YARN-6623:
--
[~eyang],
Thanks for the comments. I'm agree with #2, which we should have a way to
expose allowed whitelist and other stuffs such as allowed devices.
However, to me, limiting capability of c-e can definitely improve security. We
don't want a non-privileged user acquires excessive permissions by executing
{{c-e}} (excessive permissions like mount volumes, etc). Please let me know
your thoughts.
> Add support to turn off launching privileged containers in the
> container-executor
> -
>
> Key: YARN-6623
> URL: https://issues.apache.org/jira/browse/YARN-6623
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
>Priority: Blocker
> Attachments: YARN-6623.001.patch, YARN-6623.002.patch,
> YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch,
> YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch,
> YARN-6623.009.patch, YARN-6623.010.patch, YARN-6623.011.patch,
> YARN-6623.012.patch, YARN-6623.013.patch
>
>
> Currently, launching privileged containers is controlled by the NM. We should
> add a flag to the container-executor.cfg allowing admins to disable launching
> privileged containers at the container-executor level.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16181413#comment-16181413 ] Eric Yang commented on YARN-6623: - A couple concerns: # Moving code from Java to C without findbugs check for vulnerability, is risky for future enhancement. # Mount point white list should be placed in visible place like common-site.xml or yarn-site.xml to let other people know about the path that can be mounted. # Container-executor.cfg permission might set to 640, which prevent usability from point #2 for users. Container-executor binary is governed by setuid bits. A privileged user is allowed to do many things in Linux. Effort of trying to limit root user to less power, does not improve security. It only make system more difficult to service in situations that have yet been realized. Sorry that there are a lot of code been written for this JIRA. However, it seems a bit risky to push validation logic to root user side. It would have been better to reduce the scope of this JIRA to focus on disabling launching privileged containers on node manager side only in my opinion. The failed unit test case does not seem to be related to the latest version of patch. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch, YARN-6623.011.patch, > YARN-6623.012.patch, YARN-6623.013.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16179552#comment-16179552
]
Hadoop QA commented on YARN-6623:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 13 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
23s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
5s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated
0 new + 23 unchanged - 4 fixed = 23 total (was 27) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 74m 59s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m
35s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
18s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}142m 59s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | YARN-6623 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12888901/YARN-6623.013.patch |
| Optional Tests | asflicense compile cc mvnsite javac unit javadoc
m
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16179268#comment-16179268 ] Eric Badger commented on YARN-6623: --- Looks good to me! +1 (non-binding) pending jenkins > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch, YARN-6623.011.patch, > YARN-6623.012.patch, YARN-6623.013.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16179207#comment-16179207
]
Eric Badger commented on YARN-6623:
---
Hey [~vvasudev], thanks for the updated patch! The changes related to YARN-4266
look good. Just had one comment on the code.
{noformat}
1850 +static int set_network(const struct configuration *command_config,
1851 + const struct configuration *conf, char *out,
1852 + const size_t outlen) {
1853 +
1854 + int ret = 0;
1855 + ret = add_param_to_command_if_allowed(command_config, conf, "net",
1856 +"docker.allowed.networks",
"--net=",
1857 +0, 0, out, outlen);
1858 + if (ret != 0) {
1859 +fprintf(ERRORFILE, "Could not find requested network in allowed
networks\n");
1860 +ret = INVALID_DOCKER_NETWORK;
1861 + }
1862 + if (ret != 0) {
1863 +memset(out, 0, outlen);
1864 + }
1865 + return ret;
1866 +}
{noformat}
Something I noticed in passing. We should combine the {{if(ret != 0)}}
statements here. This same thing also occurs in {{set_capabilities()}} and
{{set_devices()}}.
> Add support to turn off launching privileged containers in the
> container-executor
> -
>
> Key: YARN-6623
> URL: https://issues.apache.org/jira/browse/YARN-6623
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
>Priority: Blocker
> Attachments: YARN-6623.001.patch, YARN-6623.002.patch,
> YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch,
> YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch,
> YARN-6623.009.patch, YARN-6623.010.patch, YARN-6623.011.patch,
> YARN-6623.012.patch
>
>
> Currently, launching privileged containers is controlled by the NM. We should
> add a flag to the container-executor.cfg allowing admins to disable launching
> privileged containers at the container-executor level.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16178317#comment-16178317
]
Hadoop QA commented on YARN-6623:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 13 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
47s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
14s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated
0 new + 23 unchanged - 4 fixed = 23 total (was 27) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 74m 52s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m
18s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
16s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}147m 26s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | YARN-6623 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12888749/YARN-6623.012.patch |
| Optional Tests | asflicense compile cc mvnsite javac unit javadoc
m
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16178263#comment-16178263
]
Hadoop QA commented on YARN-6623:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 13 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
3s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
17s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 6m 30s{color} |
{color:red} hadoop-yarn-project_hadoop-yarn generated 2 new + 0 unchanged - 0
fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated
0 new + 23 unchanged - 4 fixed = 23 total (was 27) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 74m 48s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 14m 29s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
24s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
44s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}149m 54s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
| |
hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestDockerContainerRuntime
|
| |
hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestDockerContainerRuntime
|
\\
\\
|| Subsystem || Report/
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16177577#comment-16177577 ] Varun Vasudev commented on YARN-6623: - [~andrew.wang] - YARN-4266 adds a new field for the Docker invocation so I need to update the patch to add it in. [~djp] - the discussion made sense to me. Please go ahead and file that JIRA. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16177578#comment-16177578 ] Varun Vasudev commented on YARN-6623: - [~andrew.wang] - YARN-4266 adds a new field for the Docker invocation so I need to update the patch to add it in. [~djp] - the discussion made sense to me. Please go ahead and file that JIRA. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16177381#comment-16177381 ] Junping Du commented on YARN-6623: -- Hi [~vvasudev], do you agree with what we discussed in security alias thread that only fix a small (but very important) issue instead of backporting whole YARN-6623? If so, I will go ahead to file a new jira to unblock 2.8.2 release? > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16176868#comment-16176868 ] Andrew Wang commented on YARN-6623: --- Varun, can we commit the trunk/branch-3.0 patch today? YARN-4266 is already in trunk/branch-3.0. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16176435#comment-16176435 ] Arun Suresh commented on YARN-6623: --- Thanks for the heads up [~djp].. Kindly do push this into branch-2 ass well > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16176346#comment-16176346 ] Varun Vasudev commented on YARN-6623: - [~leftnoteasy] - please don't commit the patch tomorrow. I need to incorporate YARN-4266 into this. I should have a patch for you over the weekend. Thanks! > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175908#comment-16175908 ] Junping Du commented on YARN-6623: -- bq. I plan to commit the patch to trunk/branch-3.0 tomorrow if no more opposite opinions. I think branch-2 also need this as all docker related patch should also get landed on branch-2. CC [~asuresh]. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175755#comment-16175755 ] Wangda Tan commented on YARN-6623: -- Thanks [~vvasudev] and all other folks for the hard work to finish / review the patch. I just checked the latest patch, it looks fine and all new added config knobs are properly designed. So I'm +1 to the latest patch, it looks like this patch needs rebase on top of YARN-7034. Is there any other comments? I plan to commit the patch to trunk/branch-3.0 tomorrow if no more opposite opinions. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175416#comment-16175416 ] Eric Badger commented on YARN-6623: --- Given that docker is alpha in 2.8.2, I agree with [~vinodkv] that we should not backport this to 2.8.2, as it makes significant changes to the container-executor. Nobody should even attempt seriously running docker on 2.8.2 without backporting most, if not everything that's in 2.9 and doing a large security review over their configurations and setup. So, in my mind, some docker code may be "in" 2.8.2, but it really isn't supported at all until 2.9+. The only reason I can think of to put this in 2.8.2 would be to get the changes to container-executor, but it's not obvious to me why we would need those. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175386#comment-16175386 ] Junping Du commented on YARN-6623: -- bq. We have already documented that the docker feature is alpha, not for production use (and documenting more). Given this, I don't think we should add more risk to 2.8.2. That's also my initial thinking, but [~shaneku...@gmail.com] convinced me offline that this is important for 2.8.2 even as an alpha feature - indeed still alpha for 2.9 and 3.0 and seems to affect non-docker container runtime. So I change my mind to support this backport. [~vvasudev], [~shaneku...@gmail.com] and [~miklos.szeg...@cloudera.com], what do you guys think? > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175308#comment-16175308 ] Vinod Kumar Vavilapalli commented on YARN-6623: --- [~djp], this is too big a change and a patch to put into 2.8.2. We have already documented that the docker feature is alpha, not for production use (and documenting more). Given this, I don't think we should add more risk to 2.8.2. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175249#comment-16175249 ] Miklos Szegedi commented on YARN-6623: -- Indeed, I verified and we still need the strlen in the first case, and I do not see an overflow possibility of tmp_buffer_2, so those should be okay. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175201#comment-16175201
]
Varun Vasudev commented on YARN-6623:
-
{noformat}
How would you detect the condition where the buffer doesn't have enough
size?
You copy at most bufflen-strlen(buff) characters including \0. As I said only
one strlen is enough in this case.
{noformat}
I think I understand what you're saying. The current implementation checks if
the buffer has enough space to hold the final string and will return an error
if there isn't enough space. Without the additional strlen, how would I check
that the buffer can fit the additional string?
{noformat}
381 quote_and_append_arg(&tmp_buffer, &tmp_buffer_size, " ", image_name);
That space might need to be added to the quote_and_append_arg function for
safety reasons.
I didn't get this. Can you please explain?
When we add a new arg then the space should be added by default in
quote_and_append_arg every time.
{noformat}
Ah I see what you're saying. Thanks for the explanation. That line of code got
removed in the latest patch to address the feedback from
[~shaneku...@gmail.com] in his review.
{noformat}
Is there any benefit to strcpy + strcat?
There is no need to do an strlen. Was the intention maybe to bound by the size
of tmp_buffer_2?
{noformat}
Yep. The thinking was to limit it to the size of tmp_buffer_2.
> Add support to turn off launching privileged containers in the
> container-executor
> -
>
> Key: YARN-6623
> URL: https://issues.apache.org/jira/browse/YARN-6623
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
>Priority: Blocker
> Attachments: YARN-6623.001.patch, YARN-6623.002.patch,
> YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch,
> YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch,
> YARN-6623.009.patch, YARN-6623.010.patch
>
>
> Currently, launching privileged containers is controlled by the NM. We should
> add a flag to the container-executor.cfg allowing admins to disable launching
> privileged containers at the container-executor level.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175181#comment-16175181 ] Varun Vasudev commented on YARN-6623: - [~djp] - makes sense to push it into 2.8.2; [~andrew.wang] - thanks for updating the target versions. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16173952#comment-16173952
]
Miklos Szegedi commented on YARN-6623:
--
[~vvasudev], thank you for the updated patch. I will review it but before that
let's address your questions.
bq. How would you detect the condition where the buffer doesn't have enough
size?
You copy at most bufflen-strlen(buff) characters including {{\0}}. As I said
only one strlen is enough in this case.
bq. I didn't quite understand this. What would the len do? Your understanding
is correct, we're checking if the left device is allowed.
Never mind, I was just trying to replace {{tmp_ptr - values\[i\]}} with a
length.
bq. 381 quote_and_append_arg(&tmp_buffer, &tmp_buffer_size, " ", image_name);
bq. That space might need to be added to the quote_and_append_arg function for
safety reasons.
bq. I didn't get this. Can you please explain?
When we add a new arg then the space should be added by default in
quote_and_append_arg every time.
bq. Is there any benefit to strcpy + strcat?
There is no need to do an strlen. Was the intention maybe to bound by the size
of tmp_buffer_2?
> Add support to turn off launching privileged containers in the
> container-executor
> -
>
> Key: YARN-6623
> URL: https://issues.apache.org/jira/browse/YARN-6623
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
>Priority: Blocker
> Attachments: YARN-6623.001.patch, YARN-6623.002.patch,
> YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch,
> YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch,
> YARN-6623.009.patch, YARN-6623.010.patch
>
>
> Currently, launching privileged containers is controlled by the NM. We should
> add a flag to the container-executor.cfg allowing admins to disable launching
> privileged containers at the container-executor level.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16173856#comment-16173856
]
Hadoop QA commented on YARN-6623:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 13 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
55s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
52s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
14s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
20s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 8m 20s{color} |
{color:red} hadoop-yarn-project_hadoop-yarn generated 2 new + 0 unchanged - 0
fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated
0 new + 23 unchanged - 4 fixed = 23 total (was 27) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 84m 39s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 15m
21s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}165m 36s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
| Timed out junit tests |
org.apache.hadoop.yarn.server.resourcemanager.TestRMStoreCommands |
| | org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore
|
| |
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA |
| |
org.apache.hadoop.yarn.server.resourcemanager.TestReservationSystemWithRMHA |
\\
\\
|| Subsyst
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16173795#comment-16173795 ] Junping Du commented on YARN-6623: -- Thanks [~vvasudev] for working hard on this effort and everyone for review! Shall we backport this to branch-2.8 as well, otherwise it could be a security hole for 2.8.2 as a production release? > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16170280#comment-16170280 ] Varun Vasudev commented on YARN-6623: - Thanks for the feedback [~miklos.szeg...@cloudera.com], [~shaneku...@gmail.com]. I should have a patch addressing you reviews soon. [~miklos.szeg...@cloudera.com] bq. I have some comments but before that, does the jira target 3.0-beta1? I would like to get it into beta1. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169141#comment-16169141
]
Shane Kumpf commented on YARN-6623:
---
Thanks for the patch [~vvasudev]! Overall it looks great. Thanks for the
effort. Couple of comments.
Can we print out the mount that was requested but failed for the read-only and
read write checks? I found it a bit difficult to determine which mount was the
cause of the failure. Doing the same for the devices option would be helpful as
well. Something like the following:
{code}
if (permitted_rw == 0) {
fprintf(ERRORFILE, "Requested rw mount not found in the rw whitelist
'%s', realpath=%s\n", values[i], mount_src);
ret = INVALID_DOCKER_RW_MOUNT;
goto free_and_exit;
-snip-
if (ro != 0 && permitted_ro == 0 && permitted_rw == 0) {
fprintf(ERRORFILE, "Requested ro mount not found in the ro whitelist
'%s', realpath=%s\n", values[i], mount_src);
ret = INVALID_DOCKER_RO_MOUNT;
goto free_and_exit;
}
{code}
Currently only {{docker run}} works with this patch. I know you are aware of
this. There are a few regressions in this patch from what was fixed in
YARN-6726. It would be great if we could try to address them here. I'll list
the ones I'm aware of below.
1) get_docker_inspect_command uses validate_container_id directly, instead of
validate_container_name.
{code}
if (container_name == NULL || validate_container_id(container_name) != 0) {
return INVALID_DOCKER_CONTAINER_NAME;
}
{code}
2) The {{--format}} option for {{docker inspect}} can't be quoted, or it breaks
the returned output.
{code}
quote_and_append_arg(&tmp_buffer, &tmp_buffer_size, " --format=", format);
{code}
3) The container name as the final argument can't be quoted. This happens in a
number of places.
{code}
quote_and_append_arg(&tmp_buffer, &tmp_buffer_size, "", container_name);
{code}
See this comment for what I had found previously.
[YARN-6726
Items|https://issues.apache.org/jira/browse/YARN-6726?focusedCommentId=16082179&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16082179]
> Add support to turn off launching privileged containers in the
> container-executor
> -
>
> Key: YARN-6623
> URL: https://issues.apache.org/jira/browse/YARN-6623
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
>Priority: Blocker
> Attachments: YARN-6623.001.patch, YARN-6623.002.patch,
> YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch,
> YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch,
> YARN-6623.009.patch
>
>
> Currently, launching privileged containers is controlled by the NM. We should
> add a flag to the container-executor.cfg allowing admins to disable launching
> privileged containers at the container-executor level.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16165527#comment-16165527
]
Miklos Szegedi commented on YARN-6623:
--
Thank you for the patch [~vvasudev] and for the reviews [~leftnoteasy] and
[~ebadger].
Sorry about the delay, I had a chance to look at the latest patch now. I have
some comments but before that, does the jira target 3.0-beta1?
{code}
84 printWriter.println(" " + entry.getKey() + "=" + StringUtils
85 .join(",", entry.getValue()));
{code}
writeCommandToTempFile: entry.getKey() can still contain an = in the latest
patch, which is an issue. It probably needs to be filtered in
addCommandArguments.
{code}
701char *get_config_path(const char *argv0) {
702 char *executable_file = get_executable((char *) argv0);
703 if (!executable_file) {
704fprintf(ERRORFILE, "realpath of executable: %s\n",
705errno != 0 ? strerror(errno) : "unknown");
706return NULL;
707 }
{code}
It is probably a good idea to check for {{executable_file\[0\] != 0}} as well
{code}
1150 size_t command_size = MIN(sysconf(_SC_ARG_MAX), 128*1024);
1151 char *buffer = alloc_and_clear_memory(command_size, sizeof(char));
1152 ret = get_docker_command(command_file, &CFG, buffer,
EXECUTOR_PATH_MAX);
{code}
The code passes in a different size than the actual size of the buffer.
{code}
157inline void* alloc_and_clear_memory(size_t num, size_t size) {
158 void *ret = calloc(num, size);
159 if (ret == NULL) {
160exit(OUT_OF_MEMORY);
161 }
162 return ret;
163}
{code}
It might be a good idea to print an error message here.
{code}
42static int add_to_buffer(char *buff, const size_t bufflen, const char
*string) {
{code}
Why do not you use strncat inside? It would spare one of the strlen’s.
{code}
105if(prefix != 0) {
106 tmp_ptr = strchr(values[i], prefix);
107 if (tmp_ptr == NULL) {
...
{code}
This feels a little bit less readable. I would suggest having a len instead of
tmp_ptr defaulted to strlen(tmp_ptr); Also, am I right that we are checking
only if the left device is allowed?
{code}
150 if (ret != 0) {
151memset(out, 0, outlen);
152 }
{code}
out\[0\]=0 should be sufficient, if outlen > 0 and ret != 0
{code}
162 if (0 == strncmp(container_name, CONTAINER_NAME_PREFIX,
strlen(CONTAINER_NAME_PREFIX))) {
{code}
There is no need of an strlen here, sizeof is sufficient and calculates compile
time
{code}
283 ret = add_docker_config_param(&command_config, out, outlen);
284 if (ret != 0) {
285 return BUFFER_TOO_SMALL;
{code}
Container name is not freed.
{code}
330 if (ret != 0) {
331 return BUFFER_TOO_SMALL;
332 }
{code}
Image name is not freed.
{code}
381 quote_and_append_arg(&tmp_buffer, &tmp_buffer_size, " ", image_name);
{code}
That space might need to be added to the quote_and_append_arg function for
safety reasons.
{code}
564 * 2. If the path is a directory, add a '/' at the end( if not present)
{code}
There is a small typo here.
{code}
585 if (len <= 0) {
586 return NULL;
587 }
{code}
There is a missing free here
{code}
731 strncpy(tmp_buffer_2, values[i], strlen(values[i]));
732 strncpy(tmp_buffer_2 + strlen(values[i]), ro_suffix, strlen(ro_suffix));
{code}
Why do you use strncpy here? Why not strcpy and strcat?
{code}
739 memset(tmp_buffer, 0, tmp_buffer_size);
{code}
Clearing the first character should be sufficient.
> Add support to turn off launching privileged containers in the
> container-executor
> -
>
> Key: YARN-6623
> URL: https://issues.apache.org/jira/browse/YARN-6623
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
>Priority: Blocker
> Attachments: YARN-6623.001.patch, YARN-6623.002.patch,
> YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch,
> YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch,
> YARN-6623.009.patch
>
>
> Currently, launching privileged containers is controlled by the NM. We should
> add a flag to the container-executor.cfg allowing admins to disable launching
> privileged containers at the container-executor level.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16157393#comment-16157393 ] Eric Badger commented on YARN-6623: --- I think we may want to look further into the issue with the c99 vs. c++0x compilers, but that is separate from this JIRA. I'd like to give my non-binding +1 for patch .009. Clearly we need a committer to give a binding +1, but I also think it would be good to get another pair of eyes on this since it is a large patch and because it has some potential security implications as it deals with the container-executor directly. Thanks, [~vvasudev] for all of your hard work on this! > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16157236#comment-16157236
]
Hadoop QA commented on YARN-6623:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 13 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
22s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
41s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
49s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated
0 new + 23 unchanged - 4 fixed = 23 total (was 27) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 72m 41s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
45s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}138m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | YARN-6623 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12885822/YARN-6623.009.patch |
| Optional Tests | asflicense compile cc mvnsite javac unit javadoc
mvninstall findbugs checkstyle |
| uname | Linux dcd75cc21772 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16157134#comment-16157134
]
Eric Badger commented on YARN-6623:
---
{quote}
Additionally, the native code doesn't compile for me on my rhel 6.8 VM. It
looks like it's complaining about the C99 syntax that Miklos Szegedi mentioned
here.
Weird. I've changed the initialization, should be fixed in the latest patch.
{quote}
I'm wondering why this wasn't caught by hadoopqa though. How was it that it
compiled fine for hadoopqa and failed on my box? Different compliers/versions
maybe? Not sure why that should matter since My rhel VM is using gcc 4.4.7 and
fails to compile. My mac, however, is using Apple LLVM version 8.1.0 and
compiles fine.
> Add support to turn off launching privileged containers in the
> container-executor
> -
>
> Key: YARN-6623
> URL: https://issues.apache.org/jira/browse/YARN-6623
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
>Priority: Blocker
> Attachments: YARN-6623.001.patch, YARN-6623.002.patch,
> YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch,
> YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch,
> YARN-6623.009.patch
>
>
> Currently, launching privileged containers is controlled by the NM. We should
> add a flag to the container-executor.cfg allowing admins to disable launching
> privileged containers at the container-executor level.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16155591#comment-16155591
]
Eric Badger commented on YARN-6623:
---
{{git apply}} is failing for me on DockerClient.java. It looks like it doesn't
do fuzzing like patch does. Not sure if we need to fix that or not.
Additionally, the code doesn't compile for me on my rhel 6.8 VM. It looks like
it's complaining about the C99 syntax that [~miklos.szeg...@cloudera.com]
mentioned
[here|https://issues.apache.org/jira/browse/YARN-6033?focusedCommentId=16111913&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16111913].
{noformat}
[WARNING] /usr/bin/c++
-I/home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src
-I/home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native
-isystem
/home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/../../../../../hadoop-common-project/hadoop-common/src/main/native/gtest/include
-I/home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor
-I/home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl
-fstack-check -g -O2 -Wall -pthread -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE
-o
CMakeFiles/cetest.dir/main/native/container-executor/test/utils/test_docker_util.cc.o
-c
/home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc
[WARNING] make[2]: Leaving directory
`/home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native'
[WARNING] make[1]: Leaving directory
`/home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native'
[WARNING] In file included from
/home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc:24:
[WARNING]
/home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:
In function ‘int get_docker_command(const char*, const configuration*, char*,
size_t)’:
[WARNING]
/home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:234:
error: expected primary-expression before ‘.’ token
[WARNING]
/home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:234:
error: expected primary-expression before ‘.’ token
[WARNING]
/home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:
In function ‘int get_docker_inspect_command(const char*, const configuration*,
char*, size_t)’:
[WARNING]
/home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:264:
error: expected primary-expression before ‘.’ token
[WARNING]
/home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:264:
error: expected primary-expression before ‘.’ token
[WARNING]
/home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:
In function ‘int get_docker_load_command(const char*, const configuration*,
char*, size_t)’:
[WARNING]
/home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:316:
error: expected primary-expression before ‘.’ token
[WARNING]
/home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:316:
error: expected primary-expression before ‘.’ token
[WARNING]
/home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:
In function ‘int get_docker_pull_command(const char*, const configuration*,
char*, size_t)’:
[WARNING]
/home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:360:
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16152960#comment-16152960
]
Hadoop QA commented on YARN-6623:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 13 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
18s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
49s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 1 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
50s{color} | {color:red} hadoop-yarn in trunk failed. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated
0 new + 23 unchanged - 4 fixed = 23 total (was 27) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
45s{color} | {color:red} hadoop-yarn in the patch failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 74m 46s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
46s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}147m 56s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | YARN-6623 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12885276/YARN-6623.008.patch |
| Optional Tests | asflicense findbugs xml compile cc
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16147810#comment-16147810
]
Hadoop QA commented on YARN-6623:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 13 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
2s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
47s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
12s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 6m 10s{color} |
{color:red} hadoop-yarn-project_hadoop-yarn generated 1 new + 0 unchanged - 0
fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated
0 new + 23 unchanged - 4 fixed = 23 total (was 27) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
54s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 72m 31s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
35s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}143m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issu
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16147713#comment-16147713
]
Eric Badger commented on YARN-6623:
---
Hey [~vvasudev], thanks for the update. It looks like most of my first round
comments were lost, so posting them again.
{noformat}
int is_docker_support_enabled() {
-return is_feature_enabled(DOCKER_SUPPORT_ENABLED_KEY,
-DEFAULT_DOCKER_SUPPORT_ENABLED, &executor_cfg);
+ return is_feature_enabled(DOCKER_SUPPORT_ENABLED_KEY,
+DEFAULT_DOCKER_SUPPORT_ENABLED, &executor_cfg)
+ || docker_module_enabled(&CFG);
{noformat}
The indentation here should be fixed.
{noformat}
+# The configs below deal with settings for Docker
+#[docker]
+# module.enabled=## enable/disable the module. set to "true" to enable,
disabled by default
+# docker.binary=/usr/bin/docker
+# allowed.capabilities=## comma seperated capabilities that can be granted,
e.g
CHOWN,DAC_OVERRIDE,FSETID,FOWNER,MKNOD,NET_RAW,SETGID,SETUID,SETFCAP,SETPCAP,NET_BIND_SERVICE,SYS_CHROOT,KILL,AUDIT_WRITE
+# allowed.devices=## comma seperated list of devices that can be mounted into
a container
+# allowed.networks=## comma seperated networks that can be used. e.g
bridge,host,none
+# allowed.ro-mounts=## comma seperated volumes that can be mounted as
read-only
+# allowed.rw-mounts=## comma seperate volumes that can be mounted as
read-write, add the yarn local and log dirs to this list to run Hadoop jobs
{noformat}
For the Docker configs, I think it'd be more clear if they were prefixed with
docker.
{noformat}
+
+
+
+
+
+
{noformat}
Since we're going to remove the hard-coded string in YARN-6968, should we leave
the findbugs warning and take care of it there?
{noformat}
+ protected final void addCommandArguments(String key, String value) {
+if (commandArguments.containsKey(key)) {
+ commandArguments.get(key).add(value);
+ return;
+}
{noformat}
No need to call contains() and then call get(). We can just call get and then
add the value if the return value isn't null.
{noformat}
+ @Override
+ public String toString() {
+StringBuffer ret = new StringBuffer("");
+ret.append(this.command);
{noformat}
Any reason not to create the string with the command in the constructor?
Something like
{{StringBuffer ret = new StringBuffer(this.command);}}
> Add support to turn off launching privileged containers in the
> container-executor
> -
>
> Key: YARN-6623
> URL: https://issues.apache.org/jira/browse/YARN-6623
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
>Priority: Blocker
> Attachments: YARN-6623.001.patch, YARN-6623.002.patch,
> YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch,
> YARN-6623.006.patch, YARN-6623.007.patch
>
>
> Currently, launching privileged containers is controlled by the NM. We should
> add a flag to the container-executor.cfg allowing admins to disable launching
> privileged containers at the container-executor level.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16144207#comment-16144207
]
Wangda Tan commented on YARN-6623:
--
Thanks [~vvasudev],
Apologize for the delays. I haven't done any detailed reviews yet, I will leave
detailed reviews to other folks due to my availability.
Just one quick comment: for feature enable/disable. I suggest to put under
section.
For GPU feature (YARN-6852), I made config like:
{code}
[gpu]
# Enable / disable the module
module.enabled=true/false
{code}
If you're OK with the style, I suggest to make it consistent: You can use
{{module-configs.h}} to call {{module_enabled}} and get the config.
And what is feature {{tc}}? Could you make the feature name more descriptive?
> Add support to turn off launching privileged containers in the
> container-executor
> -
>
> Key: YARN-6623
> URL: https://issues.apache.org/jira/browse/YARN-6623
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
>Priority: Blocker
> Attachments: YARN-6623.001.patch, YARN-6623.002.patch,
> YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch,
> YARN-6623.006.patch
>
>
> Currently, launching privileged containers is controlled by the NM. We should
> add a flag to the container-executor.cfg allowing admins to disable launching
> privileged containers at the container-executor level.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16142372#comment-16142372
]
Eric Badger commented on YARN-6623:
---
{noformat}
container-executor.c:run_docker()
+ docker_command = construct_docker_command(command_file);
+ docker_binary = get_docker_binary(&CFG);
I don’t see these getting freed. Am I missing this invocation somewhere?
container-executor.c:run_docker()
We call the execvp function, the running program will get replaced by the
docker invocation.
{noformat}
Ahhh yes of course. Disregard my comment
{noformat}
if (command == NULL || (strcmp(command, docker_command) != 0)) {
ret = INCORRECT_COMMAND;
}
Is command guaranteed to be null-terminated? It comes from the configuration
file, which is a Java creation and I don’t think Java null-terminates. This
comment is probably relevant for quite a few other places that are doing string
operations. We need to be very safe about this or else we risk possibly
overrunning into random regions of the heap. A safe alternative would be to use
the “n” version of all the string operations. This patch uses a mixed bag of
the regular versions and their accompanying “n” versions. I don’t quite
understand the reasoning behind the usage of each. If we guarantee that the
string is null terminated (and always null terminated) then we don’t need the
“n” versions. But even if we guarantee that the input string is null
terminated, it may accidentally have the null character chopped off by an off
by 1 error in a strdup or something like that. So my preference here would be
to use the “n” versions of all of the string functions. Thoughts?
command is guaranteed to be null terminated by the confiugration functions. If
we use the 'n' versions of the function we end up doing a 'begins with' match
instead of an exact match which could cause problems(e.g. "inspect" would match
"inspectcommand")
{noformat}
Yea I guess that makes sense. I still think we should be consistent on the
usage of the str vs strn functions, though. There are still places in the patch
using both, seemingly interchangeably.
{noformat}
docker-util.c: get_docker_load_command()
+ image_name = get_configuration_value("image", DOCKER_COMMAND_FILE_SECTION,
&command_config);
+ if (image_name == NULL) {
+return INVALID_DOCKER_IMAGE_NAME;
+ }
+
+ memset(out, 0, outlen);
Maybe being paranoid is a good thing, but get_docker_load_command and any of
the other commands from get_docker_command are always passed in a buffer that
has already been nulled out, so the memset here is redundant. I imagine it will
be similar in the other get_*_comand functions. Maybe we should put a comment
saying that it is the caller’s responsibility to null out the buffer that they
pass in and get rid of these memsets? Either that or keep the memset here and
get rid of the resets higher up in get_docker_command?
I've gotten rid of the memset in the higher calls and left the memset in the
get_docker_* functions. I just feel more comfortable having the memset there.
{noformat}
That sounds good to me
{noformat}
docker-util.c: get_docker_load_command()
+ image_name = get_configuration_value("image", DOCKER_COMMAND_FILE_SECTION,
&command_config);
+ if (image_name == NULL) {
+return INVALID_DOCKER_IMAGE_NAME;
+ }
We should validate the image name here as we do in get_docker_pull_command
below.
You can pass tar files to docker load so we can't do much validation here.
Docker save lets you save the image to any file you want and then you load it
using docker load.
{noformat}
Ah that’s true, good point.
{noformat}
docker-util.c
+static int add_param_to_command(const struct configuration *command_config,
const char *key, const char *param,
+ const int with_argument, char *out, const
size_t outlen) {
+ size_t tmp_buffer_size = 1024;
What’s the reasoning behind the buffer size here? It’s hardcoded to 1024 in a
bunch of places. This doesn’t follow the system path limit, the
EXECUTOR_PATH_MAX or the system argument length.
It's just a temporary buffer. The size doesn't matter too much because
quote_and_append_arg will resize the buffer if it's too small.
{noformat}
That makes sense. Shouldn’t we bound the size that {{quote_and_append()}} can
{{realloc()}} to, though? I think the proper size would be {{MIN(_SC_ARG_MAX,
128*1024)}}
{noformat}
docker-util.c:set_network()
+ } else if (value == NULL) {
+ret = 0;
+goto free_and_exit;
Since we didn’t find any network in the configuration, we didn’t set a network.
That seems like it should print out an error and/or fail. Right now it will
just silently do nothing.
You can launch docker containers with no network specified. It'll just pick up
the default network.
{noformat}
Ah ok. No problem then
{noformat}
docker-util.c:check_mount_permitted()
+// directory check
+
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16140761#comment-16140761
]
Hadoop QA commented on YARN-6623:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 13 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
33s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
49s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
13s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
6s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 57s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 23 new + 23 unchanged - 4 fixed = 46 total (was 27) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
56s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 80m 58s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 14m 3s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}153m 26s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
| | hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
| | TEST-cetest |
| | TEST-cetest |
\\
\\
|| Subsystem || Report/Notes ||
| D
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16137515#comment-16137515
]
Eric Badger commented on YARN-6623:
---
Hey [~vvasudev], I’ve finished my comments on the production code portion of
the patch. This doesn’t include test code. I think I’ll wait for your response
to this before I review the tests. I tried to be quite thorough and so I have
quite a few comments. I hope that they are all correct interpretations, but
it’s quite a big patch so I might have misunderstood some stuff and/or missed
some stuff.
Should we fix the no newline at the end of file warnings? The apply tool
complains about them.
{noformat:title=DockerCommand:getCommandArguments()}
public Map> getCommandArguments() {
return Collections.unmodifiableMap(commandArguments);
}
{noformat}
This will return the command as well as the arguments. Unless we are
considering the {{/usr/docker}} to be the actual command and {{inspect}} to be
one of the arguments. If that’s what we’re expecting to happen here, then the
name is a little bit misleading. This might be more of a problem with how
{{commandArguments}} is named than how this function is named.
{noformat:title=container-executor.c:construct_docker_command()}
+char *construct_docker_command(const char *command_file) {
+ int ret = 0;
+ char *buffer = (char *) malloc(EXECUTOR_PATH_MAX * sizeof(char));
{noformat}
This should use {{_SC_ARG_MAX}} as we did in YARN-6988
{{size_t command_size = MIN(sysconf(_SC_ARG_MAX), 128*1024);}}
Also, why not use {{calloc()}} instead of {{malloc()}} and then {{memset()}}?
{noformat:title=container-executor.c:run_docker()}
+ docker_command = construct_docker_command(command_file);
+ docker_binary = get_docker_binary(&CFG);
{noformat}
I don’t see these getting freed. Am I missing this invocation somewhere?
{noformat:title=container-executor.c:run_docker()}
+ memset(docker_command_with_binary, 0, EXECUTOR_PATH_MAX);
{noformat}
Is this necessary? We allocate the memory with {{calloc()}} which already
clears all of the memory upon allocation.
{noformat:title=container-executor.h}
// get the executable's filename
char* get_executable(char *argv0);
{noformat}
Do we need this declaration (in container-executor.h) since we have moved that
declaration into get_executable.h? We should also add get_executable.h in the
appropriate places. Looks like main.c and test-container-executor.c both call
{{get_executable}}.
{noformat:title=main.c:assert_valid_setup()}
-fprintf(ERRORFILE,"realpath of executable: %s\n",
- errno != 0 ? strerror(errno) : "unknown");
-flush_and_close_log_files();
-exit(-1);
+fprintf(ERRORFILE, "realpath of executable: %s\n",
+errno != 0 ? strerror(errno) : "unknown");
+exit(INVALID_CONFIG_FILE);
{noformat}
Why don’t we want to flush the log files anymore?
{noformat:title=util.c:alloc_and_clear_memory()}
+void* alloc_and_clear_memory(size_t num, size_t size) {
+ void *ret = calloc(num, size);
+ if (ret == NULL) {
+exit(OUT_OF_MEMORY);
+ }
+ return ret;
+}
{noformat}
Should we inline this? Also, if we’re going to use this function, then all
calloc calls should be replaced with this (like the ones I mentioned above)
{noformat:title=util.h}
// DOCKER_CONTAINER_NAME_INVALID = 41,
{noformat}
Should add {{(NOT USED)}} to follow convention
{noformat:title=docker-util.c:read_and_verify_command_file()}
if (command == NULL || (strcmp(command, docker_command) != 0)) {
ret = INCORRECT_COMMAND;
}
{noformat}
Is {{command}} guaranteed to be null-terminated? It comes from the
configuration file, which is a Java creation and I don’t think Java
null-terminates. This comment is probably relevant for quite a few other places
that are doing string operations. We need to be very safe about this or else we
risk possibly overrunning into random regions of the heap. A safe alternative
would be to use the “n” version of all the string operations. This patch uses a
mixed bag of the regular versions and their accompanying “n” versions. I don’t
quite understand the reasoning behind the usage of each. If we guarantee that
the string is null terminated (and always null terminated) then we don’t need
the “n” versions. But even if we guarantee that the input string is null
terminated, it may accidentally have the null character chopped off by an off
by 1 error in a strdup or something like that. So my preference here would be
to use the “n” versions of all of the string functions. Thoughts?
{noformat:title=docker-util.c:read_and_verify_command_file()}
+ if (current_len + string_len < bufflen - 1) {
+strncpy(buff + current_len, string, bufflen - current_len);
+buff[current_len + string_len] = '\0';
+return 0;
+ }
{noformat}
Here it is copying over {{bufflen - current_len}} bytes, when we really o
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16135420#comment-16135420
]
Hadoop QA commented on YARN-6623:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 13 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
7s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
51s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
23s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
5s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m
44s{color} | {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 1m 44s{color} |
{color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 44s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 49s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 1 new + 24 unchanged - 3 fixed = 25 total (was 27) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
51s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 62m 17s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 53s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}116m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6623 |
| JIRA Patch URL |
https://iss
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16135306#comment-16135306
]
Eric Badger commented on YARN-6623:
---
[~vvasudev], I started reviewing the .003 patch, but I see that you've updated
a few times since then. So I'll post my comments so far and then continue my
review with the most recent patch.
Ran test-container-executor on both my mac and on rhel6 and it passed.
1. Should {{feature.docker.enabled}} be moved to be with the rest of the docker
configs?
2. For the Docker configs, I think it'd be more clear if they were prefixed
with {{docker.}}
3. Since we're going to remove the hard-coded string in YARN-6968, should we
leave the findbugs warning and take care of it there?
4.
{noformat}
+ /**
+ * Add command commandWithArguments - this method is only meant for use by
+ * sub-classes.
+ *
{noformat}
This is no longer just adding a command. Looks like it adds arguments to the
command and adds the command as well if it doesn't exist. So the comment should
be updated.
5.
{noformat}
+this.commandArguments = new TreeMap<>();
+commandArguments.put(dockerCommandKey, new ArrayList<>());
+commandArguments.get(dockerCommandKey).add(command);
{noformat}
I’m not sure I understand how this piece of code is supposed to work. In this
method we add a new list with the key “docker-command”. But then in
{{addCommandArguments()}} we add arguments via the {{key}}, which is always
called in the subclasses with the name of the command being run (e.g. inspect).
So the map for each {{DockerCommand}} will usually have 2 keys,
“docker-command” and the command that actually needs arguments “inspect”. Is
this how it’s intended to work or did I miss something here?
6.
{noformat}
+ protected final void addCommandArguments(String key, String value) {
+if (commandArguments.containsKey(key)) {
+ commandArguments.get(key).add(value);
+ return;
+}
{noformat}
No need to call {{contains()}} and then call {{get()}}. We can just call get
and then add the value if the return value isn't null.
7.
{noformat}
+ @Override
+ public String toString() {
+StringBuffer ret = new StringBuffer("");
+ret.append(this.command);
{noformat}
Any reason not to create the string with the command in the constructor?
Something like
{{StringBuffer ret = new StringBuffer(this.command);}}
> Add support to turn off launching privileged containers in the
> container-executor
> -
>
> Key: YARN-6623
> URL: https://issues.apache.org/jira/browse/YARN-6623
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
>Priority: Blocker
> Attachments: YARN-6623.001.patch, YARN-6623.002.patch,
> YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch
>
>
> Currently, launching privileged containers is controlled by the NM. We should
> add a flag to the container-executor.cfg allowing admins to disable launching
> privileged containers at the container-executor level.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16135245#comment-16135245
]
Varun Vasudev commented on YARN-6623:
-
{quote}
#define EXECUTOR_PATH_MAX 131072
This is lots of allocation. The OS actually needs to zero all the allocated
memory before giving it to other processes after close, so this might add to
the overall CPU usage and memory bandwidth.
Given that yarn local dirs are bind mounted into containers, using 4096 has
proven problematic with even a small number of yarn local dirs/disks (3 in the
case I saw). We're going to need to make the size of this buffer configurable.
If it seems more appropriate to do so in a separate issue, I'm fine with that.
{quote}
This got fixed by YARN-6988.
> Add support to turn off launching privileged containers in the
> container-executor
> -
>
> Key: YARN-6623
> URL: https://issues.apache.org/jira/browse/YARN-6623
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
>Priority: Blocker
> Attachments: YARN-6623.001.patch, YARN-6623.002.patch,
> YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch
>
>
> Currently, launching privileged containers is controlled by the NM. We should
> add a flag to the container-executor.cfg allowing admins to disable launching
> privileged containers at the container-executor level.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16135223#comment-16135223
]
Shane Kumpf commented on YARN-6623:
---
{quote}
#define EXECUTOR_PATH_MAX 131072
This is lots of allocation. The OS actually needs to zero all the allocated
memory before giving it to other processes after close, so this might add to
the overall CPU usage and memory bandwidth.
{quote}
Given that yarn local dirs are bind mounted into containers, using 4096 has
proven problematic with even a small number of yarn local dirs/disks (3 in the
case I saw). We're going to need to make the size of this buffer configurable.
If it seems more appropriate to do so in a separate issue, I'm fine with that.
{quote}
Image can be specified by a digest, which is more secure. I do not see that
supported by the regex. IMAGE[:TAG|@DIGEST]
{quote}
This is the same regex used on the java side, just duplicated in c-e to address
the concerns with direct invocation. We should fix this both places to support
the digest notation. Again, this seems like a separate issue from what this
patch is addressing.
> Add support to turn off launching privileged containers in the
> container-executor
> -
>
> Key: YARN-6623
> URL: https://issues.apache.org/jira/browse/YARN-6623
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
>Priority: Blocker
> Attachments: YARN-6623.001.patch, YARN-6623.002.patch,
> YARN-6623.003.patch, YARN-6623.004.patch
>
>
> Currently, launching privileged containers is controlled by the NM. We should
> add a flag to the container-executor.cfg allowing admins to disable launching
> privileged containers at the container-executor level.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16135193#comment-16135193
]
Hadoop QA commented on YARN-6623:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} YARN-6623 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-6623 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12882875/YARN-6623.004.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/17031/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add support to turn off launching privileged containers in the
> container-executor
> -
>
> Key: YARN-6623
> URL: https://issues.apache.org/jira/browse/YARN-6623
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
>Priority: Blocker
> Attachments: YARN-6623.001.patch, YARN-6623.002.patch,
> YARN-6623.003.patch, YARN-6623.004.patch
>
>
> Currently, launching privileged containers is controlled by the NM. We should
> add a flag to the container-executor.cfg allowing admins to disable launching
> privileged containers at the container-executor level.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131651#comment-16131651
]
Miklos Szegedi commented on YARN-6623:
--
Thank you for the patch, [~vvasudev]. I had time to check only the first half
so far. I hope this helps.
{code}
/proc/mounts,/sys/fs/cgroup are always in the same place
{code}
This is actually not completely true. If you run in a container,/sys/fs/cgroup
can be anywhere
{code}
SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -std=c++11")
{code}
Not sure, but this might not build on old OS like centos6
{code}
490.addReadOnlyMountLocation(CGROUPS_ROOT_DIRECTORY,
491CGROUPS_ROOT_DIRECTORY, false);
{code}
This is actually a security issue. As opposed to lxcfs, mounting cgroups will
expose information about all the other containers to each container. This
change is big enough but you might want to whitelist this in the future.
{code}
76 printWriter.println("[docker-command-execution]");
77 for (Map.Entry> entry :
cmd.getCommandArguments()
78 .entrySet()) {
79printWriter.println(" " + entry.getKey() + "=" + StringUtils
80.join(",", entry.getValue()));
81 }
{code}
Probably it does worth to check, if entry for example contains “abc=\ndef” to
avoid injection attacks.
{code}
169 ret[i + 1] = '\0';
{code}
I think it is safe to do a single {{ret[I] = 0;}} after the loop
{code}
177void quote_and_append_arg(char **str, size_t *size, const char* param,
const char *arg) {
178 char *tmp = escape_single_quote(arg);
179 strcat(*str, param);
180 strcat(*str, "'");
181 if(strlen(*str) + strlen(tmp) > *size) {
182*str = (char *) realloc(*str, strlen(*str) + strlen(tmp) + 1024);
183if(*str == NULL) {
184 exit(OUT_OF_MEMORY);
185}
186*size = strlen(*str) + strlen(tmp) + 1024;
187 }
188 strcat(*str, tmp);
189 strcat(*str, "' ");
190 free(tmp);
191}
{code}
What is 1024? How about setting * size before *str, so that there is no need
for duplication?
{code}
#define EXECUTOR_PATH_MAX 131072
{code}
This is lots of allocation. The OS actually needs to zero all the allocated
memory before giving it to other processes after close, so this might add to
the overall CPU usage and memory bandwidth.
{code}
35if (command != NULL) {
36 free(command);
37}
{code}
This is not necessary, free always ignores NULL argument.
{code}
47 if (current_len + string_len < bufflen) {
{code}
bufflen-1 to allocate space for the terminating 0
{code}
48strncpy(buff + current_len, string, bufflen - current_len);
{code}
strncpy does not add 0 terminator at the end of the target, if
strlen(string)==bufflen - current_len resulting in a read buffer overflow later.
{code}
# docker.binary=/bin/docker
115 docker_binary = strdup("/usr/bin/docker”);
{code}
We should choose one and use it everywhere.
{code}
165 if (0 == strncmp(container_name, CONTAINER_NAME_PREFIX,
strlen(CONTAINER_NAME_PREFIX))) {
{code}
This is a double scan. You can just use strcmp with the same effect.
{code}
249 const char *regex_str =
"^(([a-zA-Z0-9.-]+)(:[0-9]+)?/)?([a-z0-9_./-]+)(:[a-zA-Z0-9_.-]+)?$”;
{code}
Image can be specified by a digest, which is more secure. I do not see that
supported by the regex. IMAGE[:TAG|@DIGEST]
{code}
477permitted_mount_len = strlen(permitted_mounts[i]);
478if (permitted_mounts[i][permitted_mount_len - 1] == '/') {
{code}
Buffer underflow at permitted_mount_len == 0
{code}
488static char* normalize_mount(const char* mount) {
{code}
It would be really nice to have some comments here.
{code}
503 size_t len = strlen(real_mount);
504 if (real_mount[len - 1] != '/') {
505ret_ptr = (char *) alloc_and_clear_memory(len + 2, sizeof(char));
506strncpy(ret_ptr, real_mount, len);
507ret_ptr[len] = '/';
508ret_ptr[len + 1] = '\0';
{code}
Potential buffer underflow moreover most likely the character does not match
and we end with a normalized mount path of “/“. (!)
continued...
> Add support to turn off launching privileged containers in the
> container-executor
> -
>
> Key: YARN-6623
> URL: https://issues.apache.org/jira/browse/YARN-6623
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
>Priority: Blocker
> Attachments: YARN-6623.001.patch, YARN-6623.002.patch,
> YARN-6623.003.patch
>
>
> Currently, launching privileged containers is controlled by the NM. We should
> add a flag to the container-executor.cfg allowing admins to disa
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16128525#comment-16128525
]
Hadoop QA commented on YARN-6623:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 13 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
39s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
1s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
47s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 7m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated
0 new + 25 unchanged - 2 fixed = 25 total (was 27) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
18s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 57s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m
36s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}163m 19s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
| Timed out junit tests |
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Ima
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16127536#comment-16127536
]
Hadoop QA commented on YARN-6623:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 13 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
55s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
31s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
49s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
4s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
41s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 56s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 6 new + 25 unchanged - 2 fixed = 31 total (was 27) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
52s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 72m 12s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 14m 30s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}143m 21s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | TEST-cetest |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
| | TEST-cetest |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6623 |
| JIRA Patch UR
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16127494#comment-16127494 ] Eric Badger commented on YARN-6623: --- I'll review the patch sometime this week > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16126228#comment-16126228
]
Hadoop QA commented on YARN-6623:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 13 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
53s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
33s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
45s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
0s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
37s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 54s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 6 new + 25 unchanged - 2 fixed = 31 total (was 27) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
0s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
5s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 75m 59s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 13m 25s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
25s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}146m 16s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| |
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.docker.DockerCommand.toString()
concatenates strings using + in a loop At DockerCommand.java:in a loop At
DockerCommand.java:[line 87] |
| | Unread field:field be
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116129#comment-16116129 ] Varun Vasudev commented on YARN-6623: - To fix this, we need support for sections provided by YARN-6033. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16112271#comment-16112271 ] Varun Vasudev commented on YARN-6623: - Just to note - we should document this in the release notes since it'll break backwards compatibility(if we turn off all runtime except the process runtime by default). > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16111776#comment-16111776 ] Miklos Szegedi commented on YARN-6623: -- I updated this to blocker, because I think 3.0 should not ship without this Jira. Let me know, if you have any concerns. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16111499#comment-16111499
]
Miklos Szegedi commented on YARN-6623:
--
[~dan...@cloudera.com], this is needed I think for defense in depth.
container-executor.cfg is enforced to be runnable only by root. yarn-site.xml
is not. Also container-executor does not allow now to launch something
impersonating root. This feature should be followed by the Docker code as well.
{code}
/**
* Is the user a real user account?
* Checks:
* 1. Not root
* 2. UID is above the minimum configured.
* 3. Not in banned user list
* Returns NULL on failure
*/
struct passwd* check_user(const char *user) {
{code}
Let's assume someone allows the container-executor executed from yarn but set
user to root (or run privileged docker). In this case the point running YARN as
yarn and not root is lost.
> Add support to turn off launching privileged containers in the
> container-executor
> -
>
> Key: YARN-6623
> URL: https://issues.apache.org/jira/browse/YARN-6623
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
>
> Currently, launching privileged containers is controlled by the NM. We should
> add a flag to the container-executor.cfg allowing admins to disable launching
> privileged containers at the container-executor level.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16015619#comment-16015619 ] Daniel Templeton commented on YARN-6623: It's not obvious to me why having two ways to disable privileged containers is necessary. The container-executer.cfg also lives on the NM, so what we're saying is that we want two ways to disable privileged containers on the NM, both controlled by the administrator. Is the point to keep someone from being able to use the container-executor binary as a security exploit outside of the NM? If someone manages to gain the ability to launch the container-executor directly, privileged containers are the least of our worries. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
