[jira] [Commented] (YARN-6060) Linux container executor fails to run container on directories mounted as noexec
[ https://issues.apache.org/jira/browse/YARN-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812689#comment-15812689 ] Miklos Szegedi commented on YARN-6060: -- I opened HADOOP-13963 and YARN-6077. Since I am only familiar with Yarn, do you mind, if I pick YARN-6077? > Linux container executor fails to run container on directories mounted as > noexec > > > Key: YARN-6060 > URL: https://issues.apache.org/jira/browse/YARN-6060 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, yarn >Reporter: Miklos Szegedi > Attachments: YARN-6060.000.patch, YARN-6060.001.patch > > > If node manager directories are mounted as noexec, LCE fails with the > following error: > Launching container... > Couldn't execute the container launch file > /tmp/hadoop-/nm-local-dir/usercache//appcache/application_1483656052575_0001/container_1483656052575_0001_02_01/launch_container.sh > - Permission denied -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6060) Linux container executor fails to run container on directories mounted as noexec
[ https://issues.apache.org/jira/browse/YARN-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812488#comment-15812488 ] Allen Wittenauer commented on YARN-6060: bq. should we create a JIRA to make /bin/bash configurable in UnixShellScriptBuilder? Yes, please. /bin/bash is hard-coded in a few places in the Java code and they should all be changed to either be /usr/bin/env bash or pull from a system property/configuration entry so that the shell code can define where exactly bash is located on startup. > Linux container executor fails to run container on directories mounted as > noexec > > > Key: YARN-6060 > URL: https://issues.apache.org/jira/browse/YARN-6060 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, yarn >Reporter: Miklos Szegedi > Attachments: YARN-6060.000.patch, YARN-6060.001.patch > > > If node manager directories are mounted as noexec, LCE fails with the > following error: > Launching container... > Couldn't execute the container launch file > /tmp/hadoop-/nm-local-dir/usercache//appcache/application_1483656052575_0001/container_1483656052575_0001_02_01/launch_container.sh > - Permission denied -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6060) Linux container executor fails to run container on directories mounted as noexec
[ https://issues.apache.org/jira/browse/YARN-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812382#comment-15812382 ] Miklos Szegedi commented on YARN-6060: -- Thank you, [~vvasudev], [~aw] and [~templedf] for the comments. Since the error is a configuration error, I agree we should cancel the patch. [~aw], should we create a JIRA to make /bin/bash configurable in UnixShellScriptBuilder? > Linux container executor fails to run container on directories mounted as > noexec > > > Key: YARN-6060 > URL: https://issues.apache.org/jira/browse/YARN-6060 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, yarn >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi > Attachments: YARN-6060.000.patch, YARN-6060.001.patch > > > If node manager directories are mounted as noexec, LCE fails with the > following error: > Launching container... > Couldn't execute the container launch file > /tmp/hadoop-/nm-local-dir/usercache//appcache/application_1483656052575_0001/container_1483656052575_0001_02_01/launch_container.sh > - Permission denied -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6060) Linux container executor fails to run container on directories mounted as noexec
[ https://issues.apache.org/jira/browse/YARN-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15810878#comment-15810878 ] Varun Vasudev commented on YARN-6060: - [~miklos.szeg...@cloudera.com] - the patch attached will make debugging failures a lot harder. For example, Slider localizes the users application code and then launches the application. With your patch in place, the slider agent will get launched and then subsequent launches will fail. To make things worse, MR apps will run fine. Today, all apps will fail making it easier to debug. I agree with [~aw] - directories mounted with noexec is a bad configuration. > Linux container executor fails to run container on directories mounted as > noexec > > > Key: YARN-6060 > URL: https://issues.apache.org/jira/browse/YARN-6060 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, yarn >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi > Attachments: YARN-6060.000.patch, YARN-6060.001.patch > > > If node manager directories are mounted as noexec, LCE fails with the > following error: > Launching container... > Couldn't execute the container launch file > /tmp/hadoop-/nm-local-dir/usercache//appcache/application_1483656052575_0001/container_1483656052575_0001_02_01/launch_container.sh > - Permission denied -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6060) Linux container executor fails to run container on directories mounted as noexec
[ https://issues.apache.org/jira/browse/YARN-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15806781#comment-15806781 ] Allen Wittenauer commented on YARN-6060: bq. Do we have a bug in the existing code in this case? Yes. bq. Is this what you meant by mentioning pkgsrc? Yes, although I misspoke and should have said ports. > Linux container executor fails to run container on directories mounted as > noexec > > > Key: YARN-6060 > URL: https://issues.apache.org/jira/browse/YARN-6060 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, yarn >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi > Attachments: YARN-6060.000.patch, YARN-6060.001.patch > > > If node manager directories are mounted as noexec, LCE fails with the > following error: > Launching container... > Couldn't execute the container launch file > /tmp/hadoop-/nm-local-dir/usercache//appcache/application_1483656052575_0001/container_1483656052575_0001_02_01/launch_container.sh > - Permission denied -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6060) Linux container executor fails to run container on directories mounted as noexec
[
https://issues.apache.org/jira/browse/YARN-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15806672#comment-15806672
]
Miklos Szegedi commented on YARN-6060:
--
Thank you, [~aw]. Please, remember, I still have the {{#ifdef __linux}} set.
However, I have to refer to this code again:
{code}
public UnixShellScriptBuilder(){
line("#!/bin/bash");
line();
}
{code}
Do we have a bug in the existing code in this case? Is this what you meant by
mentioning pkgsrc?
I am assuming this could help there:
{code}
public UnixShellScriptBuilder(){
line("#!/usr/bin/env bash");
line();
}
{code}
> Linux container executor fails to run container on directories mounted as
> noexec
>
>
> Key: YARN-6060
> URL: https://issues.apache.org/jira/browse/YARN-6060
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, yarn
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
> Attachments: YARN-6060.000.patch, YARN-6060.001.patch
>
>
> If node manager directories are mounted as noexec, LCE fails with the
> following error:
> Launching container...
> Couldn't execute the container launch file
> /tmp/hadoop-/nm-local-dir/usercache//appcache/application_1483656052575_0001/container_1483656052575_0001_02_01/launch_container.sh
> - Permission denied
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6060) Linux container executor fails to run container on directories mounted as noexec
[ https://issues.apache.org/jira/browse/YARN-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15806581#comment-15806581 ] Allen Wittenauer commented on YARN-6060: No, because now you just broke FreeBSD and likely other OSes. (pkgsrc patches the broken Java code. We should really fix that for them.) > Linux container executor fails to run container on directories mounted as > noexec > > > Key: YARN-6060 > URL: https://issues.apache.org/jira/browse/YARN-6060 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, yarn >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi > Attachments: YARN-6060.000.patch, YARN-6060.001.patch > > > If node manager directories are mounted as noexec, LCE fails with the > following error: > Launching container... > Couldn't execute the container launch file > /tmp/hadoop-/nm-local-dir/usercache//appcache/application_1483656052575_0001/container_1483656052575_0001_02_01/launch_container.sh > - Permission denied -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6060) Linux container executor fails to run container on directories mounted as noexec
[
https://issues.apache.org/jira/browse/YARN-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15806445#comment-15806445
]
Miklos Szegedi commented on YARN-6060:
--
Does it help, if we use {{/bin/bash}}?
> Linux container executor fails to run container on directories mounted as
> noexec
>
>
> Key: YARN-6060
> URL: https://issues.apache.org/jira/browse/YARN-6060
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, yarn
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
> Attachments: YARN-6060.000.patch, YARN-6060.001.patch
>
>
> If node manager directories are mounted as noexec, LCE fails with the
> following error:
> Launching container...
> Couldn't execute the container launch file
> /tmp/hadoop-/nm-local-dir/usercache//appcache/application_1483656052575_0001/container_1483656052575_0001_02_01/launch_container.sh
> - Permission denied
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6060) Linux container executor fails to run container on directories mounted as noexec
[ https://issues.apache.org/jira/browse/YARN-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15806409#comment-15806409 ] Allen Wittenauer commented on YARN-6060: This is basically the point that [~templedf] was making: if yarn's PATH contains a place where files can be written, it's very easy to get root (since c-e will inherit it) to execute any program called 'bash'. > Linux container executor fails to run container on directories mounted as > noexec > > > Key: YARN-6060 > URL: https://issues.apache.org/jira/browse/YARN-6060 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, yarn >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi > Attachments: YARN-6060.000.patch, YARN-6060.001.patch > > > If node manager directories are mounted as noexec, LCE fails with the > following error: > Launching container... > Couldn't execute the container launch file > /tmp/hadoop-/nm-local-dir/usercache//appcache/application_1483656052575_0001/container_1483656052575_0001_02_01/launch_container.sh > - Permission denied -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6060) Linux container executor fails to run container on directories mounted as noexec
[ https://issues.apache.org/jira/browse/YARN-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15806353#comment-15806353 ] Miklos Szegedi commented on YARN-6060: -- Thank you for the comment, [~aw]. As I said, I agree that noexec should not be set on node manager directories. That does not mean that, if set Yarn should completely fail and not run any job. Can you be more specific on this one "huge hole on misconfigured systems"? > Linux container executor fails to run container on directories mounted as > noexec > > > Key: YARN-6060 > URL: https://issues.apache.org/jira/browse/YARN-6060 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, yarn >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi > Attachments: YARN-6060.000.patch, YARN-6060.001.patch > > > If node manager directories are mounted as noexec, LCE fails with the > following error: > Launching container... > Couldn't execute the container launch file > /tmp/hadoop-/nm-local-dir/usercache//appcache/application_1483656052575_0001/container_1483656052575_0001_02_01/launch_container.sh > - Permission denied -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6060) Linux container executor fails to run container on directories mounted as noexec
[
https://issues.apache.org/jira/browse/YARN-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15806155#comment-15806155
]
Allen Wittenauer commented on YARN-6060:
{code}
+#ifdef __linux
{code}
This looks like a vendor-ism creeping in. Various contributors do test and use
more than just Linux. (and yes, lce works just fine on them.)
I'm assuming that people are setting noexec from some false sense of security.
It's pure theatrics to say that noexec provides any sort of protection to a
system like Hadoop. Lots of ways around this, never mind that Java itself is
perfectly capable (albeit usually in crappy ways) to do just as much harm as
anything else.
At this point, I don't think this patch should go in simply because it sends
the wrong message, isn't particularly useful, and opens up a huge hole on
misconfigured systems.
> Linux container executor fails to run container on directories mounted as
> noexec
>
>
> Key: YARN-6060
> URL: https://issues.apache.org/jira/browse/YARN-6060
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, yarn
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
> Attachments: YARN-6060.000.patch, YARN-6060.001.patch
>
>
> If node manager directories are mounted as noexec, LCE fails with the
> following error:
> Launching container...
> Couldn't execute the container launch file
> /tmp/hadoop-/nm-local-dir/usercache//appcache/application_1483656052575_0001/container_1483656052575_0001_02_01/launch_container.sh
> - Permission denied
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6060) Linux container executor fails to run container on directories mounted as noexec
[
https://issues.apache.org/jira/browse/YARN-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15805693#comment-15805693
]
Hadoop QA commented on YARN-6060:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 12m
49s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6060 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12846072/YARN-6060.001.patch |
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux 307b9c140260 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 2977bc6 |
| Default Java | 1.8.0_111 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/14591/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/14591/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Linux container executor fails to run container on directories mounted as
> noexec
>
>
> Key: YARN-6060
> URL: https://issues.apache.org/jira/browse/YARN-6060
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, yarn
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
> Attachments: YARN-6060.000.patch, YARN-6060.001.patch
>
>
> If node manager directories are mounted as noexec, LCE fails with the
> following error:
> Launching container...
> Couldn't execute the container launch file
> /tmp/hadoop-/nm-local-dir/usercache//appcache/application_1483656052575_0001/container_1483656052575_0001_02_01/launch_container.sh
> - Permission denied
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6060) Linux container executor fails to run container on directories mounted as noexec
[
https://issues.apache.org/jira/browse/YARN-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15805483#comment-15805483
]
Miklos Szegedi commented on YARN-6060:
--
[~aw], I absolutely agree, theoretically Yarn supports native apps, and they
will all fail with noexec. This fix enables the mainline scenario, if the user
enabled noexec for any reason.
[~templedf], bash is hardcoded for all non-Windows scenarios in the node
manager source code:
{code}
public UnixShellScriptBuilder(){
line("#!/bin/bash");
line();
}
{code}
> Linux container executor fails to run container on directories mounted as
> noexec
>
>
> Key: YARN-6060
> URL: https://issues.apache.org/jira/browse/YARN-6060
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, yarn
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
> Attachments: YARN-6060.000.patch
>
>
> If node manager directories are mounted as noexec, LCE fails with the
> following error:
> Launching container...
> Couldn't execute the container launch file
> /tmp/hadoop-/nm-local-dir/usercache//appcache/application_1483656052575_0001/container_1483656052575_0001_02_01/launch_container.sh
> - Permission denied
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6060) Linux container executor fails to run container on directories mounted as noexec
[ https://issues.apache.org/jira/browse/YARN-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15805324#comment-15805324 ] Allen Wittenauer commented on YARN-6060: bq. though in an all-Java environment, it's something an admin could get away with (barring this issue) ... except Hadoop isn't an all-Java environment, server or client-side. This will break streaming, native code, probably parts of anything that calls Shell.java, libhadoop.so if it happens to be mounted on the same dir, probably hadoop archives-logs, etc, etc. There's also leveldbjni's wacky behavior if that code is enabled. Let's not forget native MR too. Really: configuring noexec is a very bad idea on a piece of software's that sole job is to be an execution engine. It will result in all sorts of weird and mysterious failures. It's definitely never been tested and certainly not a supported configuration. bq. First, it allows path manipulation to replace bash with something nefarious. We have this problem all over the place. But we should strive to remove them. I was thinking yesterday that we should fail daemon startup if . is in the path. bq. Second, it assumes the shell is bash. That's a safe assumption; all of Hadoop's shell code is written specifically for bash v3 and v4. I think it's listed in the pre-reqs. If it's not, it should be. > Linux container executor fails to run container on directories mounted as > noexec > > > Key: YARN-6060 > URL: https://issues.apache.org/jira/browse/YARN-6060 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, yarn >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi > Attachments: YARN-6060.000.patch > > > If node manager directories are mounted as noexec, LCE fails with the > following error: > Launching container... > Couldn't execute the container launch file > /tmp/hadoop-/nm-local-dir/usercache//appcache/application_1483656052575_0001/container_1483656052575_0001_02_01/launch_container.sh > - Permission denied -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6060) Linux container executor fails to run container on directories mounted as noexec
[ https://issues.apache.org/jira/browse/YARN-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15805250#comment-15805250 ] Daniel Templeton commented on YARN-6060: [~miklos.szeg...@cloudera.com], thanks for the patch. As a veteran of software with broader platform support, the idea of execing "bash" gives me the willies. First, it allows path manipulation to replace bash with something nefarious. Second, it assumes the shell is bash. Off the top of my head, I think our container launch scripts do require bash, but hard-coding it bugs me. Especially if you get to including a full path. > Linux container executor fails to run container on directories mounted as > noexec > > > Key: YARN-6060 > URL: https://issues.apache.org/jira/browse/YARN-6060 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, yarn >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi > Attachments: YARN-6060.000.patch > > > If node manager directories are mounted as noexec, LCE fails with the > following error: > Launching container... > Couldn't execute the container launch file > /tmp/hadoop-/nm-local-dir/usercache//appcache/application_1483656052575_0001/container_1483656052575_0001_02_01/launch_container.sh > - Permission denied -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6060) Linux container executor fails to run container on directories mounted as noexec
[ https://issues.apache.org/jira/browse/YARN-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15805234#comment-15805234 ] Daniel Templeton commented on YARN-6060: [~aw], I completely agree, though in an all-Java environment, it's something an admin could get away with (barring this issue). I don't see any harm in enabling the config in environments where it can make sense. I'm not a fan of execing a script directly anyway. > Linux container executor fails to run container on directories mounted as > noexec > > > Key: YARN-6060 > URL: https://issues.apache.org/jira/browse/YARN-6060 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, yarn >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi > Attachments: YARN-6060.000.patch > > > If node manager directories are mounted as noexec, LCE fails with the > following error: > Launching container... > Couldn't execute the container launch file > /tmp/hadoop-/nm-local-dir/usercache//appcache/application_1483656052575_0001/container_1483656052575_0001_02_01/launch_container.sh > - Permission denied -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6060) Linux container executor fails to run container on directories mounted as noexec
[ https://issues.apache.org/jira/browse/YARN-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15803807#comment-15803807 ] Allen Wittenauer commented on YARN-6060: Mounting task directories with noexec is pretty much a configuration error. Jobs can and will execute other code from there. For example, jobs that use shared libraries via distributed cache will fail to mmap() properly. > Linux container executor fails to run container on directories mounted as > noexec > > > Key: YARN-6060 > URL: https://issues.apache.org/jira/browse/YARN-6060 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, yarn >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi > Attachments: YARN-6060.000.patch > > > If node manager directories are mounted as noexec, LCE fails with the > following error: > Launching container... > Couldn't execute the container launch file > /tmp/hadoop-/nm-local-dir/usercache//appcache/application_1483656052575_0001/container_1483656052575_0001_02_01/launch_container.sh > - Permission denied -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6060) Linux container executor fails to run container on directories mounted as noexec
[ https://issues.apache.org/jira/browse/YARN-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15802958#comment-15802958 ] Miklos Szegedi commented on YARN-6060: -- test4tests: I checked the patch on a cluster with noexec on the nodemanager directories. The pi MR job failed without the fix and it ran with the fix successfully. > Linux container executor fails to run container on directories mounted as > noexec > > > Key: YARN-6060 > URL: https://issues.apache.org/jira/browse/YARN-6060 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, yarn >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi > Attachments: YARN-6060.000.patch > > > If node manager directories are mounted as noexec, LCE fails with the > following error: > Launching container... > Couldn't execute the container launch file > /tmp/hadoop-/nm-local-dir/usercache//appcache/application_1483656052575_0001/container_1483656052575_0001_02_01/launch_container.sh > - Permission denied -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6060) Linux container executor fails to run container on directories mounted as noexec
[
https://issues.apache.org/jira/browse/YARN-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15802947#comment-15802947
]
Hadoop QA commented on YARN-6060:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
35s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6060 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12845891/YARN-6060.000.patch |
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux 2327622626f9 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 02766b6 |
| Default Java | 1.8.0_111 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/14573/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/14573/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Linux container executor fails to run container on directories mounted as
> noexec
>
>
> Key: YARN-6060
> URL: https://issues.apache.org/jira/browse/YARN-6060
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, yarn
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
> Attachments: YARN-6060.000.patch
>
>
> If node manager directories are mounted as noexec, LCE fails with the
> following error:
> Launching container...
> Couldn't execute the container launch file
> /tmp/hadoop-/nm-local-dir/usercache//appcache/application_1483656052575_0001/container_1483656052575_0001_02_01/launch_container.sh
> - Permission denied
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
