[jira] [Commented] (YARN-9884) Make container-executor mount logic modular
[ https://issues.apache.org/jira/browse/YARN-9884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956347#comment-16956347 ] Eric Badger commented on YARN-9884: --- Thanks for the commit, [~eyang]! Thanks for the reviews [~Jim_Brennan] and [~ccondit]! > Make container-executor mount logic modular > --- > > Key: YARN-9884 > URL: https://issues.apache.org/jira/browse/YARN-9884 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9884.001.patch, YARN-9884.002.patch, > YARN-9884.003.patch, YARN-9884.004.patch > > > The current mount logic in the container-executor is interwined with docker. > To avoid duplicating code between docker and runc, the code should be > refactored so that both runtimes can use the same common code when possible. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9884) Make container-executor mount logic modular
[ https://issues.apache.org/jira/browse/YARN-9884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955010#comment-16955010 ] Hudson commented on YARN-9884: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17550 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17550/]) YARN-9884. Make container-executor mount logic modular(eyang: rev 72b1bed9982a3b775fa60ec2ea818d1fbed683a2) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.h * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/util.c * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/util.h * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/mount-utils.c * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/mount-utils.h * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc > Make container-executor mount logic modular > --- > > Key: YARN-9884 > URL: https://issues.apache.org/jira/browse/YARN-9884 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9884.001.patch, YARN-9884.002.patch, > YARN-9884.003.patch, YARN-9884.004.patch > > > The current mount logic in the container-executor is interwined with docker. > To avoid duplicating code between docker and runc, the code should be > refactored so that both runtimes can use the same common code when possible. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9884) Make container-executor mount logic modular
[ https://issues.apache.org/jira/browse/YARN-9884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955003#comment-16955003 ] Eric Yang commented on YARN-9884: - +1 for patch 004. Committing shortly. > Make container-executor mount logic modular > --- > > Key: YARN-9884 > URL: https://issues.apache.org/jira/browse/YARN-9884 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9884.001.patch, YARN-9884.002.patch, > YARN-9884.003.patch, YARN-9884.004.patch > > > The current mount logic in the container-executor is interwined with docker. > To avoid duplicating code between docker and runc, the code should be > refactored so that both runtimes can use the same common code when possible. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9884) Make container-executor mount logic modular
[ https://issues.apache.org/jira/browse/YARN-9884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16954993#comment-16954993 ] Eric Badger commented on YARN-9884: --- I have added this as a blocker for YARN-9561, as the most recent patch on YARN-9561 is based off of patch 004 in this JIRA. Together with YARN-9562 patch 009 and YARN-9561 patch 005 this creates a fully working RuncContainerRuntime. YARN-9884.004.patch YARN-9561.005.patch YARN-9562.009.patch YARN-9564.004.patch > Make container-executor mount logic modular > --- > > Key: YARN-9884 > URL: https://issues.apache.org/jira/browse/YARN-9884 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9884.001.patch, YARN-9884.002.patch, > YARN-9884.003.patch, YARN-9884.004.patch > > > The current mount logic in the container-executor is interwined with docker. > To avoid duplicating code between docker and runc, the code should be > refactored so that both runtimes can use the same common code when possible. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9884) Make container-executor mount logic modular
[ https://issues.apache.org/jira/browse/YARN-9884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953870#comment-16953870 ] Jim Brennan commented on YARN-9884: --- Thanks for the update [~ebadger]! I am +1 on patch 004 (non-binding). > Make container-executor mount logic modular > --- > > Key: YARN-9884 > URL: https://issues.apache.org/jira/browse/YARN-9884 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9884.001.patch, YARN-9884.002.patch, > YARN-9884.003.patch, YARN-9884.004.patch > > > The current mount logic in the container-executor is interwined with docker. > To avoid duplicating code between docker and runc, the code should be > refactored so that both runtimes can use the same common code when possible. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9884) Make container-executor mount logic modular
[
https://issues.apache.org/jira/browse/YARN-9884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953177#comment-16953177
]
Hadoop QA commented on YARN-9884:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
31m 5s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 31s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m
19s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 68m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.3 Server=19.03.3 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | YARN-9884 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12983208/YARN-9884.004.patch |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit |
| uname | Linux 5e3dfc7fbd06 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 375224e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/25005/testReport/ |
| Max. process+thread count | 448 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/25005/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Make container-executor mount logic modular
> ---
>
> Key: YARN-9884
> URL: https://issues.apache.org/jira/browse/YARN-9884
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-9884.001.patch, YARN-9884.002.patch,
> YARN-9884.003.patch, YARN-9884.004.patch
>
>
> The current mount logic in the container-executor is interwined with docker.
> To avoid duplicating code between docker and runc, the code should be
> refactored so that both runtimes can use the same common code when possible.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (YARN-9884) Make container-executor mount logic modular
[
https://issues.apache.org/jira/browse/YARN-9884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953124#comment-16953124
]
Eric Badger commented on YARN-9884:
---
Thanks for the review, [~Jim_Brennan]! I've attached patch 004 to slightly
change the logic around the mount options. Since they are an array of strings,
I thought it would be better to store them in a {{mount_options}} struct so
that they would be easier to deal with. The previous patch saved all of the
options as a string together, which only makes sense in the scope of docker
arguments.
> Make container-executor mount logic modular
> ---
>
> Key: YARN-9884
> URL: https://issues.apache.org/jira/browse/YARN-9884
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-9884.001.patch, YARN-9884.002.patch,
> YARN-9884.003.patch, YARN-9884.004.patch
>
>
> The current mount logic in the container-executor is interwined with docker.
> To avoid duplicating code between docker and runc, the code should be
> refactored so that both runtimes can use the same common code when possible.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9884) Make container-executor mount logic modular
[ https://issues.apache.org/jira/browse/YARN-9884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16951032#comment-16951032 ] Jim Brennan commented on YARN-9884: --- Thanks for the updates [~ebadger]! I am +1 (non-binding) on patch 003. Some of the *DOCKER* enum values may eventually need to be renamed when we add those features to runc, but I think it makes sense to wait until those features are implemented and change the enum names at that time. > Make container-executor mount logic modular > --- > > Key: YARN-9884 > URL: https://issues.apache.org/jira/browse/YARN-9884 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9884.001.patch, YARN-9884.002.patch, > YARN-9884.003.patch > > > The current mount logic in the container-executor is interwined with docker. > To avoid duplicating code between docker and runc, the code should be > refactored so that both runtimes can use the same common code when possible. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9884) Make container-executor mount logic modular
[
https://issues.apache.org/jira/browse/YARN-9884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949822#comment-16949822
]
Hadoop QA commented on YARN-9884:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
50s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
36m 31s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 22s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m
20s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 75m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.3 Server=19.03.3 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | YARN-9884 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12982818/YARN-9884.003.patch |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit |
| uname | Linux 9bcf0ce498ac 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c561a70 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24968/testReport/ |
| Max. process+thread count | 307 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24968/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Make container-executor mount logic modular
> ---
>
> Key: YARN-9884
> URL: https://issues.apache.org/jira/browse/YARN-9884
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-9884.001.patch, YARN-9884.002.patch,
> YARN-9884.003.patch
>
>
> The current mount logic in the container-executor is interwined with docker.
> To avoid duplicating code between docker and runc, the code should be
> refactored so that both runtimes can use the same common code when possible.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (YARN-9884) Make container-executor mount logic modular
[ https://issues.apache.org/jira/browse/YARN-9884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949790#comment-16949790 ] Eric Badger commented on YARN-9884: --- Thanks for the prompt reviews, everyone! Patch 003 combines the util and docker enum error code lists together into a single one in util.h. I left the old error codes that aren't used anymore to keep the diff smaller, but I could remove those if you're in favor. We still have about 50 error codes that we can add until we go over the 128 boundary. > Make container-executor mount logic modular > --- > > Key: YARN-9884 > URL: https://issues.apache.org/jira/browse/YARN-9884 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9884.001.patch, YARN-9884.002.patch, > YARN-9884.003.patch > > > The current mount logic in the container-executor is interwined with docker. > To avoid duplicating code between docker and runc, the code should be > refactored so that both runtimes can use the same common code when possible. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9884) Make container-executor mount logic modular
[ https://issues.apache.org/jira/browse/YARN-9884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949703#comment-16949703 ] Craig Condit commented on YARN-9884: [~ebadger], this looks pretty good for the most part. However, there's one potential big problem... exit codes outside of the range 0-127 tend to be misinterpreted by shells and other tooling. Have we verified that the upper codes are being interpreted properly? Most kernel wait() function variations truncate exit codes to 8 bits, and shells treat them as signed, where negative values indicate death by signal -- the infamous 143 exit code in Hadoop is really SIGTERM (15) + 128. > Make container-executor mount logic modular > --- > > Key: YARN-9884 > URL: https://issues.apache.org/jira/browse/YARN-9884 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9884.001.patch, YARN-9884.002.patch > > > The current mount logic in the container-executor is interwined with docker. > To avoid duplicating code between docker and runc, the code should be > refactored so that both runtimes can use the same common code when possible. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9884) Make container-executor mount logic modular
[ https://issues.apache.org/jira/browse/YARN-9884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949691#comment-16949691 ] Jim Brennan commented on YARN-9884: --- [~ebadger] good job on the re-factoring. This looks pretty good to me. I was going to comment that there are a few of the DOCKER related enum values that are no longer used, like INVALID_DOCKER_RO_MOUNT, and those should be removed. Also, I think all DOCKER-specific codes should have DOCKER in the name. I agree with [~eyang] that a single list would be even better. > Make container-executor mount logic modular > --- > > Key: YARN-9884 > URL: https://issues.apache.org/jira/browse/YARN-9884 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9884.001.patch, YARN-9884.002.patch > > > The current mount logic in the container-executor is interwined with docker. > To avoid duplicating code between docker and runc, the code should be > refactored so that both runtimes can use the same common code when possible. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9884) Make container-executor mount logic modular
[ https://issues.apache.org/jira/browse/YARN-9884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949497#comment-16949497 ] Eric Badger commented on YARN-9884: --- bq. I am wondering if we need to combine the list to prevent future conflicts of the same name? I am fine with doing that. I refactored the error codes in a way that I thought would be the least invasive to how docker currently handles error codes. But having multiple error code lists is a little bit confusing because there could be collisions with error values and you won't always know whether your error came from a generic error code enum or a docker-specific one. Something I ran into while testing was that the docker daemon will return codes in the 100 range. I had initially placed all of the docker error codes in the 100 range, but then got an error 127 from the docker daemon and realized that that would be incorrectly parsed as a docker error code. So even though the docker daemon was passing back 127, the error message you would be given would be something completely unrelated to the actual error. So all in all, I am in favor of combining the lists. I'd be happy to include that in this patch or put up a separate patch to do it. > Make container-executor mount logic modular > --- > > Key: YARN-9884 > URL: https://issues.apache.org/jira/browse/YARN-9884 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9884.001.patch, YARN-9884.002.patch > > > The current mount logic in the container-executor is interwined with docker. > To avoid duplicating code between docker and runc, the code should be > refactored so that both runtimes can use the same common code when possible. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9884) Make container-executor mount logic modular
[
https://issues.apache.org/jira/browse/YARN-9884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949080#comment-16949080
]
Hadoop QA commented on YARN-9884:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 21m
52s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
30m 29s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 12s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m
26s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 89m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.3 Server=19.03.3 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | YARN-9884 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12982725/YARN-9884.002.patch |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit |
| uname | Linux e06b4744a287 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 9c72bf4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24961/testReport/ |
| Max. process+thread count | 448 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24961/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Make container-executor mount logic modular
> ---
>
> Key: YARN-9884
> URL: https://issues.apache.org/jira/browse/YARN-9884
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-9884.001.patch, YARN-9884.002.patch
>
>
> The current mount logic in the container-executor is interwined with docker.
> To avoid duplicating code between docker and runc, the code should be
> refactored so that both runtimes can use the same common code when possible.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To
[jira] [Commented] (YARN-9884) Make container-executor mount logic modular
[ https://issues.apache.org/jira/browse/YARN-9884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949038#comment-16949038 ] Eric Yang commented on YARN-9884: - [~ebadger] Sorry, I just committed YARN-9860 which breaks this patch. Thanks for resolving the potential enum number conflict between docker_error_codes and errorcodes. I am wondering if we need to combine the list to prevent future conflicts of the same name? > Make container-executor mount logic modular > --- > > Key: YARN-9884 > URL: https://issues.apache.org/jira/browse/YARN-9884 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9884.001.patch, YARN-9884.002.patch > > > The current mount logic in the container-executor is interwined with docker. > To avoid duplicating code between docker and runc, the code should be > refactored so that both runtimes can use the same common code when possible. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9884) Make container-executor mount logic modular
[ https://issues.apache.org/jira/browse/YARN-9884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949034#comment-16949034 ] Eric Badger commented on YARN-9884: --- Looks like a conflict came up just before I put up the patch, so it conflicted. Patch 002 should be fine. > Make container-executor mount logic modular > --- > > Key: YARN-9884 > URL: https://issues.apache.org/jira/browse/YARN-9884 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9884.001.patch, YARN-9884.002.patch > > > The current mount logic in the container-executor is interwined with docker. > To avoid duplicating code between docker and runc, the code should be > refactored so that both runtimes can use the same common code when possible. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9884) Make container-executor mount logic modular
[
https://issues.apache.org/jira/browse/YARN-9884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949020#comment-16949020
]
Hadoop QA commented on YARN-9884:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color}
| {color:red} YARN-9884 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-9884 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12982719/YARN-9884.001.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24960/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Make container-executor mount logic modular
> ---
>
> Key: YARN-9884
> URL: https://issues.apache.org/jira/browse/YARN-9884
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-9884.001.patch
>
>
> The current mount logic in the container-executor is interwined with docker.
> To avoid duplicating code between docker and runc, the code should be
> refactored so that both runtimes can use the same common code when possible.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9884) Make container-executor mount logic modular
[ https://issues.apache.org/jira/browse/YARN-9884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948968#comment-16948968 ] Eric Badger commented on YARN-9884: --- [~eyang], [~Jim_Brennan], [~ccondit], could you review this when you get a chance? I believe this should be the last piece of refactoring that needs to be done. I needed to do this so that I could add the bind-mount validation checking in the runc code, which currently doesn't exist. And I don't want to duplicate the validation code for runc, so I think this is a better approach and cleans the code up a little bit. > Make container-executor mount logic modular > --- > > Key: YARN-9884 > URL: https://issues.apache.org/jira/browse/YARN-9884 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9884.001.patch > > > The current mount logic in the container-executor is interwined with docker. > To avoid duplicating code between docker and runc, the code should be > refactored so that both runtimes can use the same common code when possible. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
