[jira] [Commented] (YARN-6726) Fix issues with docker commands executed by container-executor

2017-09-12 Thread Jason Lowe (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163702#comment-16163702
 ] 

Jason Lowe commented on YARN-6726:
--

The branch-2 patch also has the off-by-one error in it, see YARN-7014.
{code}
  char* input_cpy = malloc(strlen(input));
  strcpy(input_cpy, input);
{code}

It also is calling strlen as the condition in a {{for}} loop in the all_numbers 
function.  This is going to effectively turn it into an O(N^2) operation since 
it will have to re-compute the length of the string on every iteration of the 
loop.  The result of strlen() should be cached in a local variable rather than 
called repetitively -- there's no JIT compiler here to clean things up.  Looks 
like YARN-6852 cleaned that up a bit in trunk.


> Fix issues with docker commands executed by container-executor
> --
>
> Key: YARN-6726
> URL: https://issues.apache.org/jira/browse/YARN-6726
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: YARN-6726.001.patch, YARN-6726.002.patch, 
> YARN-6726.003.patch
>
>
> docker inspect, rm, stop, etc are issued through container-executor. Commands 
> other than docker run are not functioning properly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6726) Fix issues with docker commands executed by container-executor

2017-09-12 Thread Varun Saxena (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163313#comment-16163313
 ] 

Varun Saxena commented on YARN-6726:


[~leftnoteasy], this seems to break compilation in branch-2
{noformat}
[WARNING] make[2]: Leaving directory 
`/home/root1/Projects/Branch2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native'
[WARNING] make[1]: Leaving directory 
`/home/root1/Projects/Branch2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native'
[WARNING] 
/home/root1/Projects/Branch2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/string-utils.c:
 In function ‘all_numbers’:
[WARNING] 
/home/root1/Projects/Branch2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/string-utils.c:33:3:
 error: ‘for’ loop initial declarations are only allowed in C99 mode
[WARNING]for (int i = 0; i < strlen(input); i++) {
[WARNING]^
[WARNING] 
/home/root1/Projects/Branch2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/string-utils.c:33:3:
 note: use option -std=c99 or -std=gnu99 to compile your code
[WARNING] make[2]: *** 
[CMakeFiles/container.dir/main/native/container-executor/impl/utils/string-utils.c.o]
 Error 1
[WARNING] make[2]: *** Waiting for unfinished jobs
[WARNING] 
/home/root1/Projects/Branch2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c:
 In function ‘tokenize_docker_command’:
[WARNING] 
/home/root1/Projects/Branch2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c:1193:7:
 warning: unused variable ‘c’ [-Wunused-variable]
[WARNING]int c = 0;
[WARNING]^
[WARNING] make[1]: *** [CMakeFiles/container.dir/all] Error 2
[WARNING] make: *** [all] Error 2
{noformat}

YARN-5719 does not exist in branch-2. Should it be in branch-2 too?

> Fix issues with docker commands executed by container-executor
> --
>
> Key: YARN-6726
> URL: https://issues.apache.org/jira/browse/YARN-6726
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: YARN-6726.001.patch, YARN-6726.002.patch, 
> YARN-6726.003.patch
>
>
> docker inspect, rm, stop, etc are issued through container-executor. Commands 
> other than docker run are not functioning properly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6726) Fix issues with docker commands executed by container-executor

2017-09-11 Thread Shane Kumpf (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161770#comment-16161770
 ] 

Shane Kumpf commented on YARN-6726:
---

Thanks [~wangda] [~sunilg] and [~ebadger]!

> Fix issues with docker commands executed by container-executor
> --
>
> Key: YARN-6726
> URL: https://issues.apache.org/jira/browse/YARN-6726
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: YARN-6726.001.patch, YARN-6726.002.patch, 
> YARN-6726.003.patch
>
>
> docker inspect, rm, stop, etc are issued through container-executor. Commands 
> other than docker run are not functioning properly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6726) Fix issues with docker commands executed by container-executor

2017-09-11 Thread Wangda Tan (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161747#comment-16161747
 ] 

Wangda Tan commented on YARN-6726:
--

[~ebadger] / [~shaneku...@gmail.com], I just committed the patch to branch-2 as 
well. 

> Fix issues with docker commands executed by container-executor
> --
>
> Key: YARN-6726
> URL: https://issues.apache.org/jira/browse/YARN-6726
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: YARN-6726.001.patch, YARN-6726.002.patch, 
> YARN-6726.003.patch
>
>
> docker inspect, rm, stop, etc are issued through container-executor. Commands 
> other than docker run are not functioning properly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6726) Fix issues with docker commands executed by container-executor

2017-09-11 Thread Eric Badger (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161706#comment-16161706
 ] 

Eric Badger commented on YARN-6726:
---

YARN-6804 is in branch-2 now. [~leftnoteasy], can we backport this now? I did a 
quick check and it looks like the cherry-pick is pretty clean, only a few 
import inconsistencies. 

> Fix issues with docker commands executed by container-executor
> --
>
> Key: YARN-6726
> URL: https://issues.apache.org/jira/browse/YARN-6726
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
> Fix For: 3.0.0-beta1
>
> Attachments: YARN-6726.001.patch, YARN-6726.002.patch, 
> YARN-6726.003.patch
>
>
> docker inspect, rm, stop, etc are issued through container-executor. Commands 
> other than docker run are not functioning properly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6726) Fix issues with docker commands executed by container-executor

2017-08-08 Thread Wangda Tan (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16118939#comment-16118939
 ] 

Wangda Tan commented on YARN-6726:
--

Committed to trunk, thanks [~shaneku...@gmail.com] and thanks reviews from 
[~sunilg]. 

Trying to backport to branch-2, however branch-2 needs YARN-6804, Jian 
confirmed he will backport YARN-6804, so let's wait YARN-6804 gets in before 
commit this patch.

> Fix issues with docker commands executed by container-executor
> --
>
> Key: YARN-6726
> URL: https://issues.apache.org/jira/browse/YARN-6726
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
> Fix For: 3.0.0-beta1
>
> Attachments: YARN-6726.001.patch, YARN-6726.002.patch, 
> YARN-6726.003.patch
>
>
> docker inspect, rm, stop, etc are issued through container-executor. Commands 
> other than docker run are not functioning properly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6726) Fix issues with docker commands executed by container-executor

2017-08-06 Thread Sunil G (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116146#comment-16116146
 ] 

Sunil G commented on YARN-6726:
---

Got it. Thanks [~shaneku...@gmail.com]

> Fix issues with docker commands executed by container-executor
> --
>
> Key: YARN-6726
> URL: https://issues.apache.org/jira/browse/YARN-6726
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
> Attachments: YARN-6726.001.patch, YARN-6726.002.patch, 
> YARN-6726.003.patch
>
>
> docker inspect, rm, stop, etc are issued through container-executor. Commands 
> other than docker run are not functioning properly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6726) Fix issues with docker commands executed by container-executor

2017-08-04 Thread Shane Kumpf (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16114829#comment-16114829
 ] 

Shane Kumpf commented on YARN-6726:
---

Thanks for the review, [~sunilg]. I have attached a new patch that addresses 
your comments.

{quote}
Could you please to share why we need a prefix of UTILS here, is this a 
standard. #ifndef UTILS_STRING_UTILS_H
{quote}

I don't know if it's a standard, but I've seen the convention elsewhere, and it 
aligns with YARN-6852 that Wangda had called out above. It is the relative path 
to the file (utils/strings-utils.h).

> Fix issues with docker commands executed by container-executor
> --
>
> Key: YARN-6726
> URL: https://issues.apache.org/jira/browse/YARN-6726
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
> Attachments: YARN-6726.001.patch, YARN-6726.002.patch, 
> YARN-6726.003.patch
>
>
> docker inspect, rm, stop, etc are issued through container-executor. Commands 
> other than docker run are not functioning properly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6726) Fix issues with docker commands executed by container-executor

2017-08-04 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16114746#comment-16114746
 ] 

Hadoop QA commented on YARN-6726:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
29s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  0m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m 
12s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
16s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m  2s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6726 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12880436/YARN-6726.003.patch |
| Optional Tests |  asflicense  compile  cc  mvnsite  javac  unit  |
| uname | Linux dc1077390d7d 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 02bf328 |
| Default Java | 1.8.0_131 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/16712/testReport/ |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/16712/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Fix issues with docker commands executed by container-executor
> --
>
> Key: YARN-6726
> URL: https://issues.apache.org/jira/browse/YARN-6726
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
> Attachments: YARN-6726.001.patch, YARN-6726.002.patch, 
> YARN-6726.003.patch
>
>
> docker inspect, rm, stop, etc are issued through container-executor. Commands 
> other than docker run are not functioning properly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6726) Fix issues with docker commands executed by container-executor

2017-08-04 Thread Sunil G (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16114160#comment-16114160
 ] 

Sunil G commented on YARN-6726:
---

Sorry for pitching in late here:

Some minor comments:
# It will be better if we can write to LOGFILE ot ERRORFILE regarding 
{{regex_match}} failure if any from {{validate_docker_image_name}} method. It 
could help us getting for information regarding regex failure if any.
# A suggestion. {{validate_docker_image_name}} could also take {{regex_str}} as 
input. In that case we can use this method for any future regex matching.
# validate_container_id could take const param
# I think i am missing something. Could you please to share why we need a 
prefix of UTILS here, is this a standard.  {{#ifndef _UTILS_STRING_UTILS_H_}}


> Fix issues with docker commands executed by container-executor
> --
>
> Key: YARN-6726
> URL: https://issues.apache.org/jira/browse/YARN-6726
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
> Attachments: YARN-6726.001.patch, YARN-6726.002.patch
>
>
> docker inspect, rm, stop, etc are issued through container-executor. Commands 
> other than docker run are not functioning properly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6726) Fix issues with docker commands executed by container-executor

2017-08-03 Thread Wangda Tan (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16113574#comment-16113574
 ] 

Wangda Tan commented on YARN-6726:
--

Thanks [~shaneku...@gmail.com], I also checked regex.h portability offline, 
according to http://unixpapa.com/incnote/regex.html. It should exist on most 
*NIX systems. I will commit the patch tomorrow if nobody against.

> Fix issues with docker commands executed by container-executor
> --
>
> Key: YARN-6726
> URL: https://issues.apache.org/jira/browse/YARN-6726
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
> Attachments: YARN-6726.001.patch, YARN-6726.002.patch
>
>
> docker inspect, rm, stop, etc are issued through container-executor. Commands 
> other than docker run are not functioning properly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6726) Fix issues with docker commands executed by container-executor

2017-08-03 Thread Shane Kumpf (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16112863#comment-16112863
 ] 

Shane Kumpf commented on YARN-6726:
---

/cc [~sunilg], [~chris.douglas], [~vvasudev] - any feedback here? I've 
successfully tested builds on osx and linux, but it would be great to get 
additional feedback on potential portability issues.

> Fix issues with docker commands executed by container-executor
> --
>
> Key: YARN-6726
> URL: https://issues.apache.org/jira/browse/YARN-6726
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
> Attachments: YARN-6726.001.patch, YARN-6726.002.patch
>
>
> docker inspect, rm, stop, etc are issued through container-executor. Commands 
> other than docker run are not functioning properly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6726) Fix issues with docker commands executed by container-executor

2017-07-28 Thread Wangda Tan (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105732#comment-16105732
 ] 

Wangda Tan commented on YARN-6726:
--

Thanks [~shaneku...@gmail.com], my only concern is the new added regex.h may 
cause some portability issues. I would like to hear some expert opinions before 
proceed.

+[~sunilg]/[~chris.douglas].

And could you help to do some manual tests to make sure this patch works, since 
existing unit test is not capable to show if it works end to end or not.

> Fix issues with docker commands executed by container-executor
> --
>
> Key: YARN-6726
> URL: https://issues.apache.org/jira/browse/YARN-6726
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
> Attachments: YARN-6726.001.patch, YARN-6726.002.patch
>
>
> docker inspect, rm, stop, etc are issued through container-executor. Commands 
> other than docker run are not functioning properly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6726) Fix issues with docker commands executed by container-executor

2017-07-26 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16102169#comment-16102169
 ] 

Hadoop QA commented on YARN-6726:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
36s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  0m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m 
55s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
20s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 32s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6726 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12879025/YARN-6726.002.patch |
| Optional Tests |  asflicense  compile  cc  mvnsite  javac  unit  |
| uname | Linux 1469e39a09ac 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 27a1a5f |
| Default Java | 1.8.0_131 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/16558/testReport/ |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/16558/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Fix issues with docker commands executed by container-executor
> --
>
> Key: YARN-6726
> URL: https://issues.apache.org/jira/browse/YARN-6726
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
> Attachments: YARN-6726.001.patch, YARN-6726.002.patch
>
>
> docker inspect, rm, stop, etc are issued through container-executor. Commands 
> other than docker run are not functioning properly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6726) Fix issues with docker commands executed by container-executor

2017-07-26 Thread Shane Kumpf (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16102113#comment-16102113
 ] 

Shane Kumpf commented on YARN-6726:
---

Thanks again, [~wangda]. Without the flush, the output is lost when the execvp 
is called. I'm attaching a patch that moves this logging to stderr instead so 
that we get the debug output in the case of failure. The patch also pulls in 
part of string-utils from YARN-6852 to do the container id validation as you 
suggested.

> Fix issues with docker commands executed by container-executor
> --
>
> Key: YARN-6726
> URL: https://issues.apache.org/jira/browse/YARN-6726
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
> Attachments: YARN-6726.001.patch
>
>
> docker inspect, rm, stop, etc are issued through container-executor. Commands 
> other than docker run are not functioning properly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6726) Fix issues with docker commands executed by container-executor

2017-07-25 Thread Wangda Tan (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100748#comment-16100748
 ] 

Wangda Tan commented on YARN-6726:
--

Thanks for explanations [~shaneku...@gmail.com], I still not fully 
understanding #2.

I can understand the part why command goes to STDOUT, but I'm not sure why 
removing fflush can avoid this happen. Before the patch, output of 
container-executor (in time order).
a. Using command:   (fflush)
b. Output of {{docker inspect}} 

Even if we remove fflush from (a), I'm not sure how we can avoid it occurs in 
the output since it happens before (b). 

Also, I'm wondering if this is a correct solution, because this assumes inside 
c-e, there's no message should be allowed to happen before the actual docker 
command execution. Ideally we should either redirect docker command output to a 
separate file or could possibly use named pipe.

> Fix issues with docker commands executed by container-executor
> --
>
> Key: YARN-6726
> URL: https://issues.apache.org/jira/browse/YARN-6726
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
> Attachments: YARN-6726.001.patch
>
>
> docker inspect, rm, stop, etc are issued through container-executor. Commands 
> other than docker run are not functioning properly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6726) Fix issues with docker commands executed by container-executor

2017-07-25 Thread Shane Kumpf (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100062#comment-16100062
 ] 

Shane Kumpf commented on YARN-6726:
---

Thanks for the review, [~wangda].

I will add additional validation of the container name.

Regarding #2, let me try to explain with an example.

We run {{docker inspect}} via 
{{PrivilegedOperationExecutor#executePrivilegedOperation}} and one of the 
options is to "grabOutput" from the command executed by c-e. In the {{docker 
inspect}} case, this is the IP address and hostname of the container. LOGFILE 
is set to stdout, which is where {{docker inspect}} output is "grabbed" from. 
When the fflush occurs on the LOGFILE, "grabOutput" returns not only the output 
from {{docker inspect}} but also this log entry. Note in the output below how 
the "IP" entry for the container is the log entry and not the IP address. Let 
me know if this is still unclear.

{code}
2017-07-25 13:30:03,596 DEBUG 
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending out 1 
container statuses: [ContainerStatus: [ContainerId: 
container_1500987852094_0001_01_01, ExecutionType: GUARANTEED, State: 
RUNNING, Capability: , Diagnostics: , ExitStatus: -1000, 
IP: [ Using command inspect 
--format='{{range(.NetworkSettings.Networks)}}{{.IPAddress}}, 
{{end}}{{.Config.Hostname}}' container_1500987852094_0001_01_01], Host: 
localhost.localdomain]]
{code}

> Fix issues with docker commands executed by container-executor
> --
>
> Key: YARN-6726
> URL: https://issues.apache.org/jira/browse/YARN-6726
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
> Attachments: YARN-6726.001.patch
>
>
> docker inspect, rm, stop, etc are issued through container-executor. Commands 
> other than docker run are not functioning properly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6726) Fix issues with docker commands executed by container-executor

2017-07-24 Thread Wangda Tan (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099233#comment-16099233
 ] 

Wangda Tan commented on YARN-6726:
--

Thanks [~shaneku...@gmail.com] for the patch. 

Discussed with Shane offline, in general the approach looks good, I haven't 
done detailed reviews of code yet. Few comments/questions:

1) Could we do more strict container_id checking, checking string starts with 
container_ might not be enough? Probably you can check the method 
(validate_container_id) I added to YARN-6852. Which we can avoid less malicious 
kill container, etc.

2) {{LOGFILE flush}}, I'm not quite sure about this item, could you elaborate?

3) Regarding to comment from [~chris.douglas], 

bq. We also need to prevent the yarn user from becoming root  ... 
If we can limit docker command only apply to containers launched by YARN (which 
we can use strict container_id pattern matching to identify that), it should be 
already much better than what we have today. We can implement other options 
such as enable/disable component, dynamic load libraries, etc. along with 
YARN-5673. 

> Fix issues with docker commands executed by container-executor
> --
>
> Key: YARN-6726
> URL: https://issues.apache.org/jira/browse/YARN-6726
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
> Attachments: YARN-6726.001.patch
>
>
> docker inspect, rm, stop, etc are issued through container-executor. Commands 
> other than docker run are not functioning properly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6726) Fix issues with docker commands executed by container-executor

2017-07-20 Thread Chris Douglas (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16095227#comment-16095227
 ] 

Chris Douglas commented on YARN-6726:
-

Not sure if I'll have cycles to review the patch in detail, but quickly:

bq. No user input is used, so this should be safe.
We also need to prevent the {{yarn}} user from becoming root, so we can't trust 
input to the CE even if it's filled in by the NM during container launch

> Fix issues with docker commands executed by container-executor
> --
>
> Key: YARN-6726
> URL: https://issues.apache.org/jira/browse/YARN-6726
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
> Attachments: YARN-6726.001.patch
>
>
> docker inspect, rm, stop, etc are issued through container-executor. Commands 
> other than docker run are not functioning properly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6726) Fix issues with docker commands executed by container-executor

2017-07-19 Thread Shane Kumpf (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093760#comment-16093760
 ] 

Shane Kumpf commented on YARN-6726:
---

[~vvasudev], [~wangda], [~jianhe], [~sunilg], [~chris.douglas] any feedback on 
the patch?

> Fix issues with docker commands executed by container-executor
> --
>
> Key: YARN-6726
> URL: https://issues.apache.org/jira/browse/YARN-6726
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
> Attachments: YARN-6726.001.patch
>
>
> docker inspect, rm, stop, etc are issued through container-executor. Commands 
> other than docker run are not functioning properly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6726) Fix issues with docker commands executed by container-executor

2017-07-11 Thread Shane Kumpf (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16082179#comment-16082179
 ] 

Shane Kumpf commented on YARN-6726:
---

I've uploaded a patch to address the items above and a few more. The summary of 
changes and why they are needed is as follows.

# {{docker stop, rm, kill}} - The final argument is the container name. 
Wrapping the container name in single quotes results in Docker being unable to 
find the image. i.e. {{container_xyz}} becomes {{'container_xyz'}} at 
invocation time.
# {{docker inspect}} - The --format option is used to obtain the container PID, 
hostname, and IP address. Attempting to quote the inspect option does not work. 
The quotes then become part of the output. i.e. 
{{'"'"'127.0.0.1,localhost.localdomain'"'"'}} would be returned if the --format 
option is quoted. I've skipped quoting for this option. No user input is used, 
so this should be safe.
# LOGFILE flush - The LOGFILE flush containing the command to be run is a 
problem. When PrivilegedOperationExecutor is asked to return the operations 
output, this log entry becomes part of the output, breaking commands such as 
{{docker inspect}}. Removed the flush.
# {{docker pull}} - Requires special handling. The only argument here is the 
image name, which is user supplied. Quoting the image name results in the same 
issues seen for container names in the {{docker stop, rm, kill}} case. To 
address this, if the subcommand is {{pull}} it is expected that the next 
element in the array is the image name. The image name is then validated 
against the same regex used in the runtime within Java. This does add a 
dependency on the regex library, which has portability issues. I'm curious how 
we can address the portability issue? Skipping validation on specific platforms 
seems inappropriate. I'm not sure only doing the validation on the java side is 
enough, so skipping validation entirely also seems inappropriate. Open to 
suggestions here.
# {{docker -- config=/path/to/conf/dir}} - Using a Docker client config 
requires the {{--config}} option to come before any sub commands. If the first 
element in the array is this option, pop it off the front of the array, add it 
to the output, and shift the array down. The current approach being considered 
is that the Docker client config will be localized with the application, and 
thus will already be checked to ensure it is a file. No user input will be 
allowed, only shipping the config.json to be localized.

> Fix issues with docker commands executed by container-executor
> --
>
> Key: YARN-6726
> URL: https://issues.apache.org/jira/browse/YARN-6726
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
> Attachments: YARN-6726.001.patch
>
>
> docker inspect, rm, stop, etc are issued through container-executor. Commands 
> other than docker run are not functioning properly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6726) Fix issues with docker commands executed by container-executor

2017-07-11 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16082170#comment-16082170
 ] 

Hadoop QA commented on YARN-6726:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
28s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 1s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 12m 
53s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
20s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 38s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6726 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12876612/YARN-6726.001.patch |
| Optional Tests |  asflicense  compile  cc  mvnsite  javac  unit  |
| uname | Linux e869e32c2005 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 34f113d |
| Default Java | 1.8.0_131 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/16370/testReport/ |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/16370/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Fix issues with docker commands executed by container-executor
> --
>
> Key: YARN-6726
> URL: https://issues.apache.org/jira/browse/YARN-6726
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
> Attachments: YARN-6726.001.patch
>
>
> docker inspect, rm, stop, etc are issued through container-executor. Commands 
> other than docker run are not functioning properly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org