[jira] [Commented] (YARN-6988) container-executor fails for docker when command length > 4096 B
[
https://issues.apache.org/jira/browse/YARN-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131325#comment-16131325
]
Hudson commented on YARN-6988:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12205 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/12205/])
YARN-6988. container-executor fails for docker when command length > (jlowe:
rev ab1a8ae85f8c61304a0f437cdc61cc5aeda36a4b)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/util.h
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
> container-executor fails for docker when command length > 4096 B
>
>
> Key: YARN-6988
> URL: https://issues.apache.org/jira/browse/YARN-6988
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Eric Badger
>Assignee: Eric Badger
> Fix For: 3.0.0-beta1
>
> Attachments: YARN-6988.001.patch, YARN-6988.002.patch
>
>
> {{run_docker}} and {{launch_docker_container_as_user}} allocate their command
> arrays using EXECUTOR_PATH_MAX, which is hardcoded to 4096 in
> configuration.h. Because of this, the full docker command can only be 4096
> characters. If it is longer, it will be truncated and the command will fail
> with a parsing error. Because of the bind-mounting of volumes, the arguments
> to the docker command can quickly get large. For example, I passed the 4096
> limit with an 11 disk node.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6988) container-executor fails for docker when command length > 4096 B
[
https://issues.apache.org/jira/browse/YARN-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129497#comment-16129497
]
Jason Lowe commented on YARN-6988:
--
+1 lgtm. I'll commit this tomorrow if there are no objections.
> container-executor fails for docker when command length > 4096 B
>
>
> Key: YARN-6988
> URL: https://issues.apache.org/jira/browse/YARN-6988
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: YARN-6988.001.patch, YARN-6988.002.patch
>
>
> {{run_docker}} and {{launch_docker_container_as_user}} allocate their command
> arrays using EXECUTOR_PATH_MAX, which is hardcoded to 4096 in
> configuration.h. Because of this, the full docker command can only be 4096
> characters. If it is longer, it will be truncated and the command will fail
> with a parsing error. Because of the bind-mounting of volumes, the arguments
> to the docker command can quickly get large. For example, I passed the 4096
> limit with an 11 disk node.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6988) container-executor fails for docker when command length > 4096 B
[
https://issues.apache.org/jira/browse/YARN-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129218#comment-16129218
]
Hadoop QA commented on YARN-6988:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
23s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6988 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12882175/YARN-6988.002.patch |
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux 58f440aeb142 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 588c190 |
| Default Java | 1.8.0_144 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/16935/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/16935/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> container-executor fails for docker when command length > 4096 B
>
>
> Key: YARN-6988
> URL: https://issues.apache.org/jira/browse/YARN-6988
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: YARN-6988.001.patch, YARN-6988.002.patch
>
>
> {{run_docker}} and {{launch_docker_container_as_user}} allocate their command
> arrays using EXECUTOR_PATH_MAX, which is hardcoded to 4096 in
> configuration.h. Because of this, the full docker command can only be 4096
> characters. If it is longer, it will be truncated and the command will fail
> with a parsing error. Because of the bind-mounting of volumes, the arguments
> to the docker command can quickly get large. For example, I passed the 4096
> limit with an 11 disk node.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional
[jira] [Commented] (YARN-6988) container-executor fails for docker when command length > 4096 B
[
https://issues.apache.org/jira/browse/YARN-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128325#comment-16128325
]
Varun Vasudev commented on YARN-6988:
-
Sounds good [~ebadger].
> container-executor fails for docker when command length > 4096 B
>
>
> Key: YARN-6988
> URL: https://issues.apache.org/jira/browse/YARN-6988
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: YARN-6988.001.patch
>
>
> {{run_docker}} and {{launch_docker_container_as_user}} allocate their command
> arrays using EXECUTOR_PATH_MAX, which is hardcoded to 4096 in
> configuration.h. Because of this, the full docker command can only be 4096
> characters. If it is longer, it will be truncated and the command will fail
> with a parsing error. Because of the bind-mounting of volumes, the arguments
> to the docker command can quickly get large. For example, I passed the 4096
> limit with an 11 disk node.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6988) container-executor fails for docker when command length > 4096 B
[
https://issues.apache.org/jira/browse/YARN-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127949#comment-16127949
]
Hadoop QA commented on YARN-6988:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
36s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 36s{color} |
{color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 36s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 35s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6988 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12882017/YARN-6988.001.patch |
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux dd978eebe5b9 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / d265459 |
| Default Java | 1.8.0_144 |
| compile |
https://builds.apache.org/job/PreCommit-YARN-Build/16915/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| cc |
https://builds.apache.org/job/PreCommit-YARN-Build/16915/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| javac |
https://builds.apache.org/job/PreCommit-YARN-Build/16915/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/16915/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/16915/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/16915/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> container-executor fails for docker when command length > 4096 B
>
>
> Key: YARN-6988
> URL: https://issues.apache.org/jira/browse/YARN-6988
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>
[jira] [Commented] (YARN-6988) container-executor fails for docker when command length > 4096 B
[
https://issues.apache.org/jira/browse/YARN-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127493#comment-16127493
]
Eric Badger commented on YARN-6988:
---
[~vvasudev], if we just increase the EXECUTOR_PATH_MAX then we would be wasting
a lot of memory if the system arg length was set to lower. Also, YARN-6623 is a
pretty large patch and might take some time to go through review. I would
prefer to put up a patch here, if that's alright. I can put up the patch today.
> container-executor fails for docker when command length > 4096 B
>
>
> Key: YARN-6988
> URL: https://issues.apache.org/jira/browse/YARN-6988
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Eric Badger
>Assignee: Eric Badger
>
> {{run_docker}} and {{launch_docker_container_as_user}} allocate their command
> arrays using EXECUTOR_PATH_MAX, which is hardcoded to 4096 in
> configuration.h. Because of this, the full docker command can only be 4096
> characters. If it is longer, it will be truncated and the command will fail
> with a parsing error. Because of the bind-mounting of volumes, the arguments
> to the docker command can quickly get large. For example, I passed the 4096
> limit with an 11 disk node.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6988) container-executor fails for docker when command length > 4096 B
[
https://issues.apache.org/jira/browse/YARN-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16125926#comment-16125926
]
Varun Vasudev commented on YARN-6988:
-
[~ebadger] - I'm increasing the EXECUTOR_PATH_MAX to 128K as part of YARN-6623.
Can you please take a look?
> container-executor fails for docker when command length > 4096 B
>
>
> Key: YARN-6988
> URL: https://issues.apache.org/jira/browse/YARN-6988
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Eric Badger
>Assignee: Eric Badger
>
> {{run_docker}} and {{launch_docker_container_as_user}} allocate their command
> arrays using EXECUTOR_PATH_MAX, which is hardcoded to 4096 in
> configuration.h. Because of this, the full docker command can only be 4096
> characters. If it is longer, it will be truncated and the command will fail
> with a parsing error. Because of the bind-mounting of volumes, the arguments
> to the docker command can quickly get large. For example, I passed the 4096
> limit with an 11 disk node.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6988) container-executor fails for docker when command length > 4096 B
[
https://issues.apache.org/jira/browse/YARN-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122348#comment-16122348
]
Eric Badger commented on YARN-6988:
---
We should definitely increase the limit, but don't want to malloc the entire
max amount of memory that the system allows, since that would be a waste. A
compromise could be to set the arg length to the min of the system value and
128K.
> container-executor fails for docker when command length > 4096 B
>
>
> Key: YARN-6988
> URL: https://issues.apache.org/jira/browse/YARN-6988
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Eric Badger
>Assignee: Eric Badger
>
> {{run_docker}} and {{launch_docker_container_as_user}} allocate their command
> arrays using EXECUTOR_PATH_MAX, which is hardcoded to 4096 in
> configuration.h. Because of this, the full docker command can only be 4096
> characters. If it is longer, it will be truncated and the command will fail
> with a parsing error. Because of the bind-mounting of volumes, the arguments
> to the docker command can quickly get large. For example, I passed the 4096
> limit with an 11 disk node.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
